Skip to content

Partition tables with @partition_by #36

Open

Description

@codecapo

@partition_by sets the target table's Iceberg partition spec. Split out of #10.

Crate: crates/engine, crates/storage

Done when

  • @partition_by: col and the transforms day(col), month(col), bucket(n, col) create the matching partition spec
  • Written data files are partitioned, and a filter on the partition column prunes files (test counts the files scanned)
  • Unknown columns or transforms are errors with the model's file and line
  • Changing @partition_by on an existing table is an error for now (spec evolution later)

Depends on: #10

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. It suggests that the RFC 0001 revision #73 either gives @partition_by a precise grammar or removes it while the RFC is still a Draft, and that the partitioned writes here wait until after 0.1.0, since partitioning changes which files a query reads, not its results.

    Suggested additions to the Done-when:

    • The directive accepts exactly the forms docs: revise RFC 0001 with the incremental rules (decision D2) #73 writes into the RFC, which may replace the transform list in this issue's first Done-when item. Today the RFC row allows both day and day(ordered_at), takes one value and defines no transform list or multi-field form (docs/rfcs/0001-project-files.md:120 at a0b5e6c), and its example -- @partition_by: day (:103) uses day, which is both an output column (:104) and an Iceberg transform name. If docs: revise RFC 0001 with the incremental rules (decision D2) #73 removes the directive instead, this issue waits for a later revision that adds it back, which is not a breaking change (:156-157).
    • The spec is set in create_table, which today gives Iceberg only a name, a schema and properties (crates/storage/src/lib.rs:127-131), and write_files splits each batch by partition instead of writing everything through one writer built with .build(None) (:361-362). Both change together, because a table with a spec rejects a commit of files written without partition values. The pinned iceberg-rust (Cargo.toml:29) already has the public pieces (the partition_spec field of TableCreation, RecordBatchPartitionSplitter, FanoutWriter and build(Some(partition_key))), so the change stays inside those two functions, and partitioned files still commit through append_if_head, which calls write_files (crates/storage/src/lib.rs:203) and then commits only if the head has not moved (:219). The insert_into of datafusion-iceberg is no substitute, because it commits through its own append path without the head check of ADR 0006 or the infostash. properties. Test: writing rows for two days into a table partitioned by day gives two partition directories; the pruned scan is already this issue's second Done-when item.

    Work that moved to new tickets:

    Suggested re-score: 5 stays, as an estimate. Section 8.5 of the design review of 2026-11-01 does not score the partitioned writes, which its section 8.4 postpones; the scored work that touches this issue is the grammar decision inside D2, 3 points for the whole RFC 0001 revision in #73. For the same reason, #36 could leave the 0.1.0 milestone, since the design review of 2026-11-01 counts unpartitioned tables among the limits a pre-alpha may ship with once they are documented. The move needs the 0.1.0 compiler to reject the directive with an error that names it, as suggested on #7, instead of parsing it into a map that only its own unit test reads (crates/compiler/src/lib.rs:9-16, called at :25). This is a suggestion only, and #19 decides.

    Source: design review of 2026-11-01, finding C19, its verdict, and sections 8.4 and 8.5 (decision D2).

  3. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    #101 removes @partition_by from RFC 0001 while it is a Draft: tables are created without a partition spec and the replace commit covers unpartitioned tables only. The first Done-when item here waits on the later revision that brings partitioning back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions