Repository navigation
test: split integration_datafusion_test.rs into scoped files - #88
Merged
Merged
Conversation
gabotechs
force-pushed
the
gabotechs/split-integration-tests
branch
from
October 28, 2026 11:45
f069f96 to
5dc1094
Compare
gabotechs
force-pushed
the
gabotechs/split-integration-tests
branch
from
October 28, 2026 12:22
5dc1094 to
12fce6e
Compare
gabotechs
force-pushed
the
gabotechs/split-integration-tests
branch
3 times, most recently
from
November 19, 2026 09:04
dd8b136 to
aa1f0d3
Compare
Base automatically changed from
gabotechs/iceberg-datafusion-test-tools
to
main
November 19, 2026 09:17
Move each test in the catch-all integration_datafusion_test.rs to a file for what it tests, and build its tables with FixtureCatalog instead of hand-assembling a MemoryCatalog, namespace, and TableCreation: - projection and predicate pushdown tests join projection_pushdown.rs and filter_pushdown.rs - the metadata-table schema test joins metadata_projection.rs, renamed metadata_tables.rs to cover both - catalog_provider.rs, insert_into.rs, plan_nodes.rs, and schema_evolution.rs hold the rest There's no shared test module. display_schema() joins the test tools, next to data_file_name(), since three files snapshot schemas with it. The rest is inlined: tests of the catalog provider register it in their own SessionContext, which is the API under test, and the few helpers used more than once in a file stay local to it. The old helpers dropped each TempDir right after taking its path, so every run left its tables behind in the system temp dir. FixtureCatalog owns its directory, so the tests now clean up after themselves. Assertions and snapshots are unchanged.
gabotechs
force-pushed
the
gabotechs/split-integration-tests
branch
from
November 19, 2026 09:17
aa1f0d3 to
ef892b9
Compare
gabotechs
marked this pull request as ready for review
November 19, 2026 09:24
alamb
approved these changes
Nov 19, 2026
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks like a good change to me -- thanks @gabotechs
| @@ -0,0 +1,491 @@ | |||
| // Licensed to the Apache Software Foundation (ASF) under one | |||
| // or more contributor license agreements. See the NOTICE file | |||
Contributor
There was a problem hiding this comment.
one thing we found in datafusion is that each test binary tends to be large (e.g. datafusion is 100MB) so if you make a bunch of test binaries, your total disk size is going to be large
You can get the same organizational benefits with a single test binary and a bunch of modules like
crates/datafusion/tests/integration.mod
crates/datafusion/tests/integration/insert_into.rs
...
Contributor
Author
There was a problem hiding this comment.
Indeed, this also happened in:
I'm going to do this here as well.
Thanks for the review!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #87.
integration_datafusion_test.rshad become a catch-all. This moves each of its tests to a file for what it tests, and builds their tables withFixtureCatalogfrom the test tools instead of hand-assembling aMemoryCatalog, namespace, andTableCreationin every test.test_table_projectionprojection_pushdown.rstest_table_predict_pushdownfilter_pushdown.rstest_metadata_tablemetadata_tables.rs, which ismetadata_projection.rsrenamed to cover both of its testscatalog_provider.rsinsert_into.rstest_plan_nodes_are_inspectableplan_nodes.rstest_scan_after_schema_evolution_reads_provider_columnsschema_evolution.rsThere's no shared test module.
display_schema()joins the test tools, next todata_file_name(), since three files snapshot schemas with it. The rest is inlined: tests of the catalog provider register it in their ownSessionContext, which is the API under test, and the few helpers used more than once in a file stay local to it.The insert tests still write their data with
INSERT, since the write path is what they test; only their empty tables come from the test tools.The old helpers dropped each
TempDirright after taking its path, so every run left about ten directories of tables behind in the system temp dir.FixtureCatalogowns its directory, so the tests now clean up after themselves.Assertions and snapshots are unchanged.
AGENTS.mdandREADME.mdpoint at the new files.🤖 Generated with Claude Code