Skip to content

perf: [iceberg] Remove IcebergFileStream, use iceberg-rust's parallelization, bump iceberg-rust to latest, cache SchemaAdapter #3051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mbutrovich merged 20 commits into from
Jan 20, 2026
Merged

perf: [iceberg] Remove IcebergFileStream, use iceberg-rust's parallelization, bump iceberg-rust to latest, cache SchemaAdapter #3051

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ef9af62
Fix granularity of metrics updates in IcebergFileStream.
mbutrovich Jan 6, 2026
01738dd
Stash caching the schema adapter.
mbutrovich Jan 6, 2026
1a3066c
Rely on iceberg-rust parallelization and get rid of IcebergFileStream…
mbutrovich Jan 6, 2026
62dc943
Fix machete
mbutrovich Jan 6, 2026
a5fcd0e
Update metrics test.
mbutrovich Jan 7, 2026
63a0b72
Merge branch 'main' into more_more_iceberg_file_stream
mbutrovich Jan 7, 2026
fe72369
Fix schema adapter caching issue.
mbutrovich Jan 7, 2026
b5db867
Set DataFusion's target_partitions to match Spark's spark.task.cpus.
mbutrovich Jan 9, 2026
d92c305
Merge branch 'task_cpus' into more_more_iceberg_file_stream
mbutrovich Jan 9, 2026
2ae6a0d
Set Iceberg-Rust's parallelism to match DataFusion's target_partition…
mbutrovich Jan 9, 2026
b8b8c0d
Don't busy-poll tokio stream for plans without CometScan.
mbutrovich Jan 10, 2026
f17099a
Merge branch 'busy_poll_fix' into more_more_iceberg_file_stream
mbutrovich Jan 10, 2026
eaaf173
Merge branch 'main' into more_more_iceberg_file_stream
mbutrovich Jan 12, 2026
79c3cc3
test fast path to avoid waker churn
mbutrovich Jan 13, 2026
a784a0b
test fast path to avoid waker churn
mbutrovich Jan 13, 2026
6dd69fc
Merge branch 'main' into more_more_iceberg_file_stream
mbutrovich Jan 16, 2026
77083f0
cargo update after upmerge main.
mbutrovich Jan 16, 2026
d6c5eaf
Merge branch 'main' into more_more_iceberg_file_stream
mbutrovich Jan 20, 2026
fca64bd
Bump to iceberg-rust main latest commit. Cargo update.
mbutrovich Jan 20, 2026
ab21b1e
Not sure why prettier was necessary but okay.
mbutrovich Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading