[EDsCODE]

add_files is slow when it registers many small files. In production, on a table with 28 columns and files of about 43 KB holding 200 rows each, registering 1,000 files took 19.8 s and registering 5,000 files took 104.8 s, not counting client overhead. About 63% of that was the commit, at roughly 13 ms per file; the rest was reading the files.

The commit wrote every catalog row with a statement of its own: one per data file, one per partition value and one per column statistic. With 28 columns that is about 30 round trips to the catalog database per file, which at about 0.4 ms each accounts for the 13 ms. The same code path (addDataFiles) registers the files of INSERT, CREATE TABLE AS and MERGE, which paid the same cost per file and per column.

What changed

• DuckLakeCommit.insertDataFiles registers all files of a commit. It sends the rows of ducklake_data_file, ducklake_file_partition_value and ducklake_file_column_stats as JDBC batches of up to 1,000 rows, which the driver pipelines, so a commit waits on a few round trips per thousand rows rather than one per row. The batches are built and executed inside the commit action, so a retried commit builds them again from scratch, and the transaction is unchanged.
• writeTableColumnStats takes every column at once: one batch of updates, then one batch of inserts for the columns no row was updated for. It now requires distinct columns, since every update runs before the first insert.
• Identifiers, values and the order of rows within each catalog table are unchanged. The DuckLake catalog schema has no foreign keys or triggers, so the interleaving across tables does not matter. A clash on the ducklake_data_file primary key still surfaces with SQLState 23505, which the driver's batch exception carries, and is retried as before.
• add_files read footers with at most 8 threads. A file costs a length lookup and a tail read on object storage, so the threads mostly wait: about 60 ms per file per thread in production. The limit is now ducklake.add-files-procedure.max-concurrent-file-reads, 32 by default, documented in ducklake.md. A read in flight holds a footer buffer (48 KB by default) and one object store connection, so 32 keeps about 15 concurrent calls within the default S3 pool of 500 connections, and the coordinator does little CPU work per read.
• The check for files the catalog already records compared names with = ANY(:fileNames). PostgreSQL hashes the array only in a plan made for the bound values; a generic plan, which a statement prepared on the server can switch to after a few executions on a pooled connection, compares every row with every name. The check now uses IN (SELECT unnest(:fileNames)), which selects the same rows and hashes the names in every plan. On a synthetic table of 2 million recorded files with 5,000 names, the old form took 1.2 s with a custom plan and 19 s with a generic one; the new form takes 1.3 to 1.4 s with either. The scan itself remains, since nothing indexes path, and the regular expression is unchanged.

Benchmark

BenchmarkDuckLakeAddFiles (test sources) registers copies of a 28-column, 200-row, 46 KB Parquet file through add_files, against the PostgreSQL container the tests use and files on the local file system. It splits the time the way the production measurement did: a second call with the same files reads them all and fails at the start of the commit, and the commit takes the rest. -Dround-trip=200us routes the connector's catalog connections through a proxy that delays them; that measured 430 to 470 µs per statement, close to production. Medians of the commit time:

	before	after
1,000 files, production-like latency	11.7 s	0.17 s
5,000 files, production-like latency	53.8 s	0.74 s
1,000 files, local container	2.0 s	0.13 s
5,000 files, local container	9.9 s	0.66 s

Locally the read phase takes 50 to 300 ms and says nothing about object storage. Driving the file reader through a file system that adds 30 ms per request, 1,000 files took 8.4 s with 8 threads (production: 7.5 s), 4.2 s with 16, 2.1 s with 32 and 1.1 s with 64, so reading scales with the thread count.

Projected for production at 28 columns: 1,000 files in about 1.9 s of reading and 0.2 s of commit, down from 19.8 s; 5,000 files in about 9.4 s of reading and 1 s of commit, down from 104.8 s. Reading now dominates, and raising the property shortens it further.

Tests

• TestDuckLakeDataFileRows pins every catalog row the commit writes: data files, partition values, file column statistics, name mappings, table statistics, table column statistics, the snapshot and the import receipt. One test does this for a partitioned add_files of four files that cover a shared and a separate name mapping, a file spanning partitions, NULL partition values, a missing column, NaN and unknown bounds. The other covers a partitioned INSERT, then an INSERT after ADD COLUMN that updates the statistics of some columns and creates them for another. Sizes are read from the Parquet files themselves. Both tests also check that DuckDB reads the rows and the row ids the files were given. They were written and passing before the change; swapping two bound values makes both fail.
• TestDuckLakeAddFiles.testDeleteFilesCannotBeRegistered covers the delete file branch of the recorded file check.
• The whole trino-ducklake suite passes locally with TZ=UTC: 275 tests, no failures. The -P ci checks (checkstyle, modernizer, dependency analysis, SpotBugs, PMD, Javadoc) and Error Prone pass locally.

🤖 Generated with Claude Code