You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The design review of 2026-11-01 touches this issue. It suggests that the RFC 0001 revision #73 either gives @partition_by a precise grammar or removes it while the RFC is still a Draft, and that the partitioned writes here wait until after 0.1.0, since partitioning changes which files a query reads, not its results.
Suggested additions to the Done-when:
The directive accepts exactly the forms docs: revise RFC 0001 with the incremental rules (decision D2) #73 writes into the RFC, which may replace the transform list in this issue's first Done-when item. Today the RFC row allows both day and day(ordered_at), takes one value and defines no transform list or multi-field form (docs/rfcs/0001-project-files.md:120 at a0b5e6c), and its example -- @partition_by: day (:103) uses day, which is both an output column (:104) and an Iceberg transform name. If docs: revise RFC 0001 with the incremental rules (decision D2) #73 removes the directive instead, this issue waits for a later revision that adds it back, which is not a breaking change (:156-157).
The spec is set in create_table, which today gives Iceberg only a name, a schema and properties (crates/storage/src/lib.rs:127-131), and write_files splits each batch by partition instead of writing everything through one writer built with .build(None) (:361-362). Both change together, because a table with a spec rejects a commit of files written without partition values. The pinned iceberg-rust (Cargo.toml:29) already has the public pieces (the partition_spec field of TableCreation, RecordBatchPartitionSplitter, FanoutWriter and build(Some(partition_key))), so the change stays inside those two functions, and partitioned files still commit through append_if_head, which calls write_files (crates/storage/src/lib.rs:203) and then commits only if the head has not moved (:219). The insert_into of datafusion-iceberg is no substitute, because it commits through its own append path without the head check of ADR 0006 or the infostash. properties. Test: writing rows for two days into a table partitioned by day gives two partition directories; the pruned scan is already this issue's second Done-when item.
Work that moved to new tickets:
docs: revise RFC 0001 with the incremental rules (decision D2) #73: decides in RFC 0001 whether @partition_by gets a grammar for this issue (which transforms, whether it takes several fields, and whether a bare day names a column or the transform) or is removed together with its use in the daily_revenue example.
Suggested re-score: 5 stays, as an estimate. Section 8.5 of the design review of 2026-11-01 does not score the partitioned writes, which its section 8.4 postpones; the scored work that touches this issue is the grammar decision inside D2, 3 points for the whole RFC 0001 revision in #73. For the same reason, #36 could leave the 0.1.0 milestone, since the design review of 2026-11-01 counts unpartitioned tables among the limits a pre-alpha may ship with once they are documented. The move needs the 0.1.0 compiler to reject the directive with an error that names it, as suggested on #7, instead of parsing it into a map that only its own unit test reads (crates/compiler/src/lib.rs:9-16, called at :25). This is a suggestion only, and #19 decides.
Source: design review of 2026-11-01, finding C19, its verdict, and sections 8.4 and 8.5 (decision D2).
#101 removes @partition_by from RFC 0001 while it is a Draft: tables are created without a partition spec and the replace commit covers unpartitioned tables only. The first Done-when item here waits on the later revision that brings partitioning back.
Activity
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It suggests that the RFC 0001 revision #73 either gives
@partition_bya precise grammar or removes it while the RFC is still a Draft, and that the partitioned writes here wait until after 0.1.0, since partitioning changes which files a query reads, not its results.Suggested additions to the Done-when:
dayandday(ordered_at), takes one value and defines no transform list or multi-field form (docs/rfcs/0001-project-files.md:120at a0b5e6c), and its example-- @partition_by: day(:103) usesday, which is both an output column (:104) and an Iceberg transform name. If docs: revise RFC 0001 with the incremental rules (decision D2) #73 removes the directive instead, this issue waits for a later revision that adds it back, which is not a breaking change (:156-157).create_table, which today gives Iceberg only a name, a schema and properties (crates/storage/src/lib.rs:127-131), andwrite_filessplits each batch by partition instead of writing everything through one writer built with.build(None)(:361-362). Both change together, because a table with a spec rejects a commit of files written without partition values. The pinned iceberg-rust (Cargo.toml:29) already has the public pieces (thepartition_specfield ofTableCreation,RecordBatchPartitionSplitter,FanoutWriterandbuild(Some(partition_key))), so the change stays inside those two functions, and partitioned files still commit throughappend_if_head, which callswrite_files(crates/storage/src/lib.rs:203) and then commits only if the head has not moved (:219). Theinsert_intoof datafusion-iceberg is no substitute, because it commits through its own append path without the head check of ADR 0006 or theinfostash.properties. Test: writing rows for two days into a table partitioned by day gives two partition directories; the pruned scan is already this issue's second Done-when item.Work that moved to new tickets:
@partition_bygets a grammar for this issue (which transforms, whether it takes several fields, and whether a baredaynames a column or the transform) or is removed together with its use in thedaily_revenueexample.Suggested re-score: 5 stays, as an estimate. Section 8.5 of the design review of 2026-11-01 does not score the partitioned writes, which its section 8.4 postpones; the scored work that touches this issue is the grammar decision inside D2, 3 points for the whole RFC 0001 revision in #73. For the same reason, #36 could leave the 0.1.0 milestone, since the design review of 2026-11-01 counts unpartitioned tables among the limits a pre-alpha may ship with once they are documented. The move needs the 0.1.0 compiler to reject the directive with an error that names it, as suggested on #7, instead of parsing it into a map that only its own unit test reads (
crates/compiler/src/lib.rs:9-16, called at :25). This is a suggestion only, and #19 decides.Source: design review of 2026-11-01, finding C19, its verdict, and sections 8.4 and 8.5 (decision D2).
codecapo commentedon Nov 10, 2026
#101 removes
@partition_byfrom RFC 0001 while it is a Draft: tables are created without a partition spec and the replace commit covers unpartitioned tables only. The first Done-when item here waits on the later revision that brings partitioning back.docs: revise RFC 0001 with the incremental rules (#101)