Skip to content

SQL UPDATE Support for DataFusion Integration #2205

Open

Description

@wirybeaver

What's the feature are you trying to implement?

Implement SQL UPDATE functionality for iceberg-datafusion integration, enabling row-level updates with WHERE clause filtering. This completes the essential DML operations alongside INSERT and the upcoming MERGE INTO support #2201.

Refer to the insert_into support

SQL Example

  UPDATE orders
  SET status = 'shipped', shipped_date = current_date()
  WHERE status = 'pending' AND payment_confirmed = true;

Overall Architecture

UPDATE table SET col1 = val1, col2 = val2 WHERE condition
   ↓
TableProvider::update() [NEW]
   ↓
IcebergTableScan (with WHERE filters)
   ↓
[Optional if partitioned: Project partition + Repartition + Sort for partitioned tables]
   ↓
IcebergUpdateWriteExec [NEW] - Apply assignments, write new files, track deleted
files
   ↓
CoalescePartitionsExec (reuse existing)
   ↓
IcebergUpdateCommitExec [NEW] - Commit via RowDelta transaction
   ↓
RecordBatch(count: UInt64)

Strategy: Copy-on-Write (COW)

  1. Scan table with WHERE filters to find matching rows
  2. Apply UPDATE assignments (evaluate expressions)
  3. Write modified rows to new data files
  4. Mark original files as deleted
  5. Commit atomically with RowDelta (add new files + remove old files)

The following tasks are already completed on the PoC branch with 6 commits. Will raise formal PRs one after another as the fork repo doesn't support stacking PRs.

  • feat(transaction): add RowDelta transaction action for row-level modi… #2203, shared with MERGE INTO
  • IcebergUpdateWriteExec - write phase with conditional assignment application
  • Add performance optimizations: Partition-aware updates (only scan/rewrite affected partitions) + File-level filtering using manifest statistics
  • IcebergUpdateCommitExec - commit phase using RowDelta transaction
  • TableProvider::update() trait implementation (Datafusion Hook)
  • Integration tests and validation

Willingness to contribute

I can contribute to this feature independently

Activity

  1. blackmwk commented on Mar 5, 2026

    @blackmwk
    Contributor

    Thanks @wirybeaver for creating this issue. Do datafusion support dml syntax such as update, delete now?

  2. blackmwk commented on Mar 5, 2026

    @blackmwk
    Contributor

    Seems datafusion still not supporing these statements: https://datafusion.apache.org/user-guide/sql/dml.html

  3. wirybeaver commented on Mar 5, 2026

    @wirybeaver
    ContributorAuthor
  4. changed the title [-]SQL UPDATE Support for Iceberg-DataFusion[/-] [+]SQL UPDATE Support for DataFusion Integration[/+] on Mar 5, 2026
  5. blackmwk commented on Mar 11, 2026

    @blackmwk
    Contributor

    Thanks @wirybeaver for raising this. I think it should support both mor and cow?

  6. wirybeaver commented on Mar 22, 2026

    @wirybeaver
    ContributorAuthor

    @blackmwk Check my comment under the #2269 Prefer implementing cow and mor in a incremental manner

  7. github-actions commented on Sep 19, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

  8. wirybeaver commented on Sep 19, 2026

    @wirybeaver
    ContributorAuthor

    waiting for #2620

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicEpic issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions