fix(format): preserve dialect-specific SQL in MODEL/AUDIT/METRIC headers

[mday-io]

Description

#5864 stopped formatting from breaking SQLMesh's own MODEL/AUDIT/METRIC header properties. On tsql, allow_partials TRUE was becoming (1 = 1), which doesn't parse. The fix was to stop transpiling headers entirely.

That was too broad. It also stopped transpiling header properties that really are warehouse SQL, which caused two problems:

• Warehouse types lost their dialect spelling. columns (ts DATETIME2(6)) was quietly rewritten to the generic TIMESTAMP(6). On tsql this got worse with each run: DATETIME2 → TIMESTAMP → reparsed as ROWVERSION → VARBINARY. Two sqlmesh format runs could turn a datetime column into a binary one without any error.
• List properties broke on BigQuery. Header lists like tags ['C1', 'c2'], ignored_rules [...] and grain [id] were rewritten as ARRAY(...). BigQuery reads ARRAY( as the start of a subquery, so a list with more than one value failed to load after formatting (Required keyword: 'value' missing for Property).

This PR formats each header property on its own instead of applying one rule to the whole header:

• Warehouse SQL uses the model's dialect. This covers columns, audits, physical_properties, partitioned_by, grain/grains, macro properties, and expression fields inside kind such as time_data_type, unique_key and time_column.
• SQLMesh settings stay dialect-agnostic. This covers allow_partials, description, and scalar kind properties such as forward_only.
• The split is automatic. Each property is sorted by the type declared on its field, so new properties land in the right group without extra work. If one is missed, the safe outcome is a keyword left in a generic spelling, not a user's SQL being rewritten.
• Renamed properties are covered. grain and table_properties are renamed to grains and physical_properties before validation, so the automatic split can't see them. They are mapped to the same group as the field they're renamed to.
• Lists use the dialect's own syntax. tags and ignored_rules are written as [...] for dialects that write arrays that way (BigQuery, DuckDB, Snowflake, ClickHouse, StarRocks). Every other dialect keeps ARRAY(...). This matters for tsql, SQLite and others that use [...] to quote identifiers: there, ['a', 'b'] would reload as a single identifier. The list items themselves are never converted to the model's dialect.

Applies to MODEL, AUDIT, and METRIC headers.

Related: #6035. The partitioned_by DATE_TRUNC(...) and tags/ignored_rules breakage reported there is fixed here.

Not fixed here: BigQuery JSON paths in the query body, such as JSON_VALUE(x, '$."28d_click"'), are rewritten to '$.28d_click'. That comes from sqlglot's BigQuery output (still present in sqlglot 30.19), not from header formatting, so it will be handled separately.

Test Plan

• test_format_model_expressions_meta_render_policy (tsql and fabric) and test_format_audit_expressions_meta_render_policy: columns, audits, physical_properties, and expression properties nested in kind keep their dialect spelling, while SQLMesh's scalar properties stay dialect-agnostic.
• test_format_model_expressions_is_idempotent: a second format run changes nothing. Covers columns, audits, kind, physical_properties, SQLMesh scalars, and macro properties.
• test_format_model_expressions_time_column_dialect: time_column keeps its dialect's identifier quoting (e.g. tsql [end]).
• test_format_model_expressions_macro_property_comments_preserved_with_dialect: comments inside a macro header property's arguments survive formatting, and a second run changes nothing.
• test_format_model_expressions_kind_scalar_sibling_dialect: on tsql, a kind block with both time_column and forward_only survives a round trip. After reloading through load_sql_based_model, model.kind.forward_only is True.
• test_format_model_expressions_list_property_array_literal (BigQuery, DuckDB, Snowflake × tags/ignored_rules): lists render as [...], reload with their values intact, and a second run changes nothing.
• test_format_model_expressions_list_property_bracket_identifier_dialects (tsql, SQLite × tags/ignored_rules): lists keep ARRAY(...) and reload with their values intact.
• test_format_model_expressions_array_property_no_dialect_unchanged: output is unchanged when the model has no dialect.
• test_format_model_expressions_grain_alias_render_policy and test_format_model_expressions_table_properties_alias_render_policy: grain and table_properties render in the model's dialect, and a second run changes nothing.
• test_format_bigquery_header_list_properties: runs sqlmesh format on a BigQuery project with tags, ignored_rules, grain and partitioned_by DATE_TRUNC(...), then reloads it and checks the values survive.
• pytest tests/core/test_dialect.py tests/core/test_format.py: 193 passed.
• pre-commit (ruff, ruff-format, mypy) on the changed files: clean.

Release Note

Fix: sqlmesh format corrupting MODEL/AUDIT/METRIC headers

On models with an explicit dialect, sqlmesh format could:

• Rewrite warehouse types in the header to a generic spelling. This affected types in columns, audits, physical_properties, and kind properties such as time_data_type and time_column. On tsql and fabric it got worse with each run: DATETIME2 → TIMESTAMP → VARBINARY across two runs. For an SCD Type 2 model's time_data_type, that turned the valid_from/valid_to columns from a datetime type into a binary one.
• Rewrite header lists such as tags ['a', 'b'], ignored_rules and grain to ARRAY(...). On BigQuery, a model with more than one value in such a list then failed to load.

SQLMesh's own header settings (allow_partials, description, the kind name, boolean kind properties like forward_only, etc.) are unaffected and still render dialect-agnostically.

Action for affected users: after upgrading, re-run sqlmesh format and review the diff. Types that were already rewritten will be restored to their dialect spelling. BigQuery models that failed to load because of ARRAY(...) in the header need that line changed back to [...] by hand before they load.

Checklist

• I have run make style and fixed any issues
• I have added tests for my changes (if applicable)
• All existing tests pass (make fast-test)
• My commits are signed off (git commit -s) per the DCO

[cmgoffena13]

Hey @mday-io -- should this fix #6035 as well? This one snuck by me, are we waiting on users to get back to us? Seems like an important fix

[mday-io]

@cmgoffena13 was waiting on the reporter to confirm he could test it via Slack - never got back. Seems unrelated to 6035 though

[mday-io]

@cmgoffena13 regarding 6035

This PR fixes the part of #6035 that actually breaks model loading - partitioned_by DATE_TRUNC(date, MONTH) no longer gets rewritten to the unparseable DATE_TRUNC('MONTH', date).

It doesn't cover the other symptom in that issue: ignored_rules ['nomissingaudits'] still gets reformatted to ignored_rules ARRAY('nomissingaudits') on BigQuery. That's a different code path - ignored_rules is just a set of rule-name strings, not warehouse SQL, so this PR intentionally renders it dialect-agnostically like allow_partials or description. The rewrite is cosmetic (the array still parses and evaluates the same either way), but it does mean formatting isn't idempotent for BigQuery projects using ignored_rules.

So, should have a separate fix than this PR.

[cmgoffena13]

@mday-io the issue I reported on there broke due to tags, I'd imagine ignored_rules might have the same effect. Is tags dialect-agnostic after this fix as well? If it is, it will break model loading, ARRAY() fails in the model DDL section.

[albertosuman-1k5]

Hey @mday-io
I'm planning to test the fix by the end of the week

[albertosuman-1k5]

I just tested this branch against our project (~2k models, BigQuery dialect).

sqlmesh format rewrites tags ['C1'] to tags ARRAY('C1') in MODEL headers, which then fails to load: Required keyword: 'value' missing for Property — the sqlmesh dialect parser only accepts bracket array literals. This is the issue that also @cmgoffena13 mentioned

It also rewrote a string literal inside a query body: JSON_VALUE(action, '$."28d_click"') → '$.28d_click', which is invalid BigQuery JSONPath (keys starting with digits need quoting). Both suggest header/property expressions are still being rendered with a non-BigQuery/non-sqlmesh dialect instead of preserved.

[mday-io]

Thank you very much. Looking into this ASAP.

[albertosuman-1k5]

Hi @mday-io
Any news on the fix?

[mday-io]

@albertosuman-1k5 thanks for testing on your models. I pushed a fix for the tags ['C1'] → ARRAY('C1') problem: list properties like tags, ignored_rules and grain now stay [...] on BigQuery, and the formatted models load again. There's a new test that runs sqlmesh format on a BigQuery model. Could you re-run the branch on your models?

The JSON path problem ('$."28d_click"' → '$.28d_click') comes from sqlglot, not this PR. That will need to be handled separately.

@cmgoffena13 I was wrong earlier to call the ignored_rules → ARRAY(...) rewrite cosmetic: with more than one value it broke loading on BigQuery. That's fixed now, along with tags.

[cmgoffena13]

@mday-io -- is this ready for review? I can take a look

[mday-io]

@cmgoffena13 plz

[cmgoffena13]

@mday-io -- Small comments, main one is testing that the data types don't change for regression testing against the unexpected bug. For release notes, I mentioned the first PR did introduce bugs in data types due to changing them to a different data type. If we release this, it won't format the data types back, so we should put a disclaimer on what happened and how they can manually fix it if they are affected. Something like below:

---

Fix: sqlmesh format silently changed column types in model headers (v0.236.0 and later)

Starting in v0.236.0, sqlmesh format wrote MODEL and AUDIT headers without applying the model's dialect. Some warehouse-specific types were rewritten to a spelling that the warehouse reads as a different, valid type. SQLMesh casts query results to the types declared in the header, so no error was raised. The model was rebuilt and the data was converted to the new type.

This release formats header properties that contain warehouse SQL (columns, kind properties such as time_data_type, audits, physical_properties, etc.) using the model's dialect again.

Affected types

Dialect	Declared type	Silently became	Result
T-SQL (SQL Server, Azure SQL)	NVARCHAR, NCHAR (including NVARCHAR(MAX))	VARCHAR, CHAR	Characters outside the column's code page are stored as ?
T-SQL	DATETIME2, SMALLDATETIME	VARBINARY (after two format runs)	Datetimes are stored as raw bytes
T-SQL, Fabric	ROWVERSION / TIMESTAMP	VARBINARY	Column is no longer a row version
BigQuery	DATETIME	TIMESTAMP	Local wall-clock times are stored as UTC timestamps
MySQL	LONGTEXT, MEDIUMTEXT	TEXT	Values over 64 KB are truncated or rejected, depending on sql_mode
Postgres, Redshift	REAL	DOUBLE PRECISION	Column widens from 4 to 8 bytes

This affects any header property that declares a type: columns, time_data_type on SCD Type 2 models (the valid_from and valid_to columns), and casts inside audits (...).

What to do

1. Check whether you're affected. You're affected if you ran sqlmesh format (or a pre-commit hook, or format-on-save in the UI) on v0.236.0 or later, and a model header used one of the types above. In git history, the format commit changes a header type. In sqlmesh plan, the model showed as modified even though its query didn't change.
2. Restore the original types from git. Re-running sqlmesh format won't restore them, because the header now contains a valid type and the formatter can't tell it was changed.
3. Upgrade and apply a plan. The plan detects the restored types and rebuilds the affected models. Review the backfill before you apply it. Data written while the wrong type was in place, such as Unicode text stored as ?, can only be recovered by rebuilding the model from its source.

[cmgoffena13]

We should add in a regression test for the bug that was introduced as well. Having format change the data type after multiple formats occurred. Something like this:

@pytest.mark.parametrize(
    "dialect,column_type",
    [
        ("bigquery", "DATETIME"),
        ("tsql", "NVARCHAR(100)"),
    ],
)
def test_format_model_expressions_preserves_column_types(dialect: str, column_type: str):
    """Formatting must not change a declared column type.
    Rendering these generically produced a different type that the dialect still accepts
    (BigQuery `DATETIME` -> `TIMESTAMP`, tsql `NVARCHAR` -> `VARCHAR`), so the model loaded
    without error and the physical table was rebuilt with the wrong type.
    """
    source = f"MODEL (name a.b, kind FULL, dialect {dialect}, columns (c {column_type})); SELECT 1 AS c"
    declared = load_sql_based_model(
        parse(source, default_dialect=dialect), dialect=dialect
    ).columns_to_types
    once = format_model_expressions(parse(source, default_dialect=dialect), dialect=dialect)
    twice = format_model_expressions(parse(once, default_dialect=dialect), dialect=dialect)
    assert once == twice
    model = load_sql_based_model(parse(twice, default_dialect=dialect), dialect=dialect)
    assert model.columns_to_types == declared

[mday-io]

Added in 68da7f4. test_format_model_expressions_preserves_column_types formats a BigQuery DATETIME and a T-SQL DATETIME2(6) model twice and reloads the model after each pass, asserting columns_to_types doesn't change. On main it fails the way you described (DATETIME → TIMESTAMP, DATETIME2 → ROWVERSION → VARBINARY), and it passes here. It replaces the text-only DATETIME2 idempotency case, since this one checks the loaded types.

[cmgoffena13]

Something to note here: this affects audits, so this will happen in T-SQL:

1. User writes: audits (my_audit(flag := true))
2. After format: audits (my_audit(flag := (1 = 1)))

Not sure it can't be helped, might just be a casualty of getting this to work for everything

[mday-io]

This one wasn't unavoidable: this PR caused it, since main writes flag := TRUE there. It also wasn't idempotent: the first format wrote (1 = 1) and the second split it across lines, so format --check failed once. Fixed in 68da7f4: boolean literals in dialect-rendered header values stay TRUE/FALSE. They still get transpiled when the value is used, so the audit query still renders WHERE (1 = 1) on T-SQL. I added audits (my_audit(flag := true)) to the idempotency test.

[albertosuman-1k5]

@mday-io The fix works for us!

The JSON path problem is actually not even a problem. BigQuery now accepts the field as it is without having to use quotes. We'll just remove them then.

So feel free to merge this. Thanks a lot!

[mday-io]

Thanks for the review @cmgoffena13. Both inline comments are addressed in 68da7f4, and I've added release notes to the description: the bug came in with v0.236.0 (#5864), the list of affected types, and recovery steps.

Following up on your earlier question about #6035: this PR now covers both parts of it, not just partitioned_by. tags, ignored_rules and grain stay dialect-agnostic, but list values are now written in each dialect's own array syntax. That's ['a', 'b'] for BigQuery, DuckDB and Snowflake. For T-SQL and SQLite it's ARRAY('a', 'b'), because [...] is identifier quoting there. test_format_bigquery_header_list_properties runs sqlmesh format on a BigQuery project with all of those and checks the model still loads. So I think this can close #6035.

Thanks also @albertosuman-1k5 for testing it against a real BigQuery project. @cmgoffena13, could you re-review when you get a chance?