Skip to content

test: split integration_datafusion_test.rs into scoped files - #88

Merged
gabotechs merged 2 commits into
mainfrom
gabotechs/split-integration-tests
Nov 19, 2026
Merged

gabotechs merged 2 commits into
mainfrom
gabotechs/split-integration-tests

Conversation

@gabotechs

Copy link
Copy Markdown
Contributor

Stacked on #87.

integration_datafusion_test.rs had become a catch-all. This moves each of its tests to a file for what it tests, and builds their tables with FixtureCatalog from the test tools instead of hand-assembling a MemoryCatalog, namespace, and TableCreation in every test.

Test Now in
test_table_projection projection_pushdown.rs
test_table_predict_pushdown filter_pushdown.rs
test_metadata_table metadata_tables.rs, which is metadata_projection.rs renamed to cover both of its tests
the 3 provider tests catalog_provider.rs
the 3 insert tests insert_into.rs
test_plan_nodes_are_inspectable plan_nodes.rs
test_scan_after_schema_evolution_reads_provider_columns schema_evolution.rs

There's no shared test module. display_schema() joins the test tools, next to data_file_name(), since three files snapshot schemas with it. The rest is inlined: tests of the catalog provider register it in their own SessionContext, which is the API under test, and the few helpers used more than once in a file stay local to it.

The insert tests still write their data with INSERT, since the write path is what they test; only their empty tables come from the test tools.

The old helpers dropped each TempDir right after taking its path, so every run left about ten directories of tables behind in the system temp dir. FixtureCatalog owns its directory, so the tests now clean up after themselves.

Assertions and snapshots are unchanged. AGENTS.md and README.md point at the new files.

🤖 Generated with Claude Code

@gabotechs
gabotechs added this pull request to stack #89 October 28, 2026 11:19
@gabotechs
gabotechs force-pushed the gabotechs/split-integration-tests branch from f069f96 to 5dc1094 Compare October 28, 2026 11:45
@gabotechs
gabotechs force-pushed the gabotechs/split-integration-tests branch from 5dc1094 to 12fce6e Compare October 28, 2026 12:22
@gabotechs
gabotechs force-pushed the gabotechs/split-integration-tests branch 3 times, most recently from dd8b136 to aa1f0d3 Compare November 19, 2026 09:04
Base automatically changed from gabotechs/iceberg-datafusion-test-tools to main November 19, 2026 09:17
Move each test in the catch-all integration_datafusion_test.rs to a file
for what it tests, and build its tables with FixtureCatalog instead of
hand-assembling a MemoryCatalog, namespace, and TableCreation:

- projection and predicate pushdown tests join projection_pushdown.rs
  and filter_pushdown.rs
- the metadata-table schema test joins metadata_projection.rs, renamed
  metadata_tables.rs to cover both
- catalog_provider.rs, insert_into.rs, plan_nodes.rs, and
  schema_evolution.rs hold the rest

There's no shared test module. display_schema() joins the test tools,
next to data_file_name(), since three files snapshot schemas with it.
The rest is inlined: tests of the catalog provider register it in their
own SessionContext, which is the API under test, and the few helpers
used more than once in a file stay local to it.

The old helpers dropped each TempDir right after taking its path, so
every run left its tables behind in the system temp dir. FixtureCatalog
owns its directory, so the tests now clean up after themselves.

Assertions and snapshots are unchanged.
@gabotechs
gabotechs force-pushed the gabotechs/split-integration-tests branch from aa1f0d3 to ef892b9 Compare November 19, 2026 09:17
@gabotechs
gabotechs marked this pull request as ready for review November 19, 2026 09:24

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good change to me -- thanks @gabotechs

@@ -0,0 +1,491 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one thing we found in datafusion is that each test binary tends to be large (e.g. datafusion is 100MB) so if you make a bunch of test binaries, your total disk size is going to be large

You can get the same organizational benefits with a single test binary and a bunch of modules like

crates/datafusion/tests/integration.mod
crates/datafusion/tests/integration/insert_into.rs
...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, this also happened in:

I'm going to do this here as well.

Thanks for the review!

@gabotechs
gabotechs merged commit a26f387 into main Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants