Conversation
…chema On engines whose sqlglot dialect lowercases unquoted identifiers (e.g. BigQuery, DuckDB), TableDiff.key_columns normalized `on` column names via normalize_identifiers() and then looked them up verbatim in the schema returned by adapter.columns(), which preserves the table's actual column casing. When the two disagreed, this raised a bare KeyError before any query ran (table_diff.py:326), and in the single-key path caused a similar KeyError deeper in _fetch_sample. `skip_columns` had the same normalize-then-exact-match mismatch, so columns intended to be excluded silently were not. Add TableDiff._resolve_column_name() to fall back to a case-insensitive match against the schema when there's no exact match, and use it when building key_columns (for both the single- and multi-column paths) and when resolving skip_columns. Add regression tests reproducing the reported KeyError for both single- and multi-column `on` lists, plus a case-insensitive skip_columns test. Fixes SQLMesh#6067 Signed-off-by: Vaibhav Naik <naikvaib@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6067.
Note: I originally looked at #5970 (Fabric CI failing) but found it's already fully addressed by two open, unreviewed PRs from the issue's own reporter/assignee (#6041, #6042), so I didn't duplicate that work and picked a different open bug instead.
#6067:
table_diffraisesKeyErrorwhen-o/skip_columnsnames aren't lowercase on BigQuery/DuckDB. Root cause:key_columnsnormalizes-onames via sqlglot's dialect (lowercasing on BigQuery/DuckDB) then does an exact-match lookup againstadapter.columns(), which preserves the table's real casing —skip_columnshas the identical bug.Added
TableDiff._resolve_column_name(): returns the exact schema key if present, else falls back to a case-insensitive match. Applied to both the single- and multi-columnkey_columnspaths, and convertedskip_columnsto acached_propertyresolved the same way against both source and target schemas.Added
test_data_diff_on_columns_with_non_lowercase_namesandtest_data_diff_skip_columns_with_non_lowercase_names(DuckDB repro matching the issue).Verified:
pytest tests/core/test_table_diff.py— 18/18 passed (16 pre-existing + 2 new).ruff check/ruff formatclean.mypy sqlmesh/core/table_diff.py— 0 errors in this file.