Skip to content

Commit ed30b96

Browse files
claudeclaude
committed
Merge upstream sqlmesh/sqlmesh main
Syncs this fork's main with the real upstream sqlmesh/sqlmesh main (b0862ae), which had drifted ~7 weeks / 51 commits behind. Preserves this fork's own pyOpenSSL pin commit (68d4cc1).
2 parents 7b7e539 + b0862ae commit ed30b96

83 files changed

Lines changed: 5838 additions & 370 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,8 @@ install-dev-dbt-%:
6363
fi; \
6464
if [ "$$version" = "1.3.0" ]; then \
6565
echo "Applying overrides for dbt $$version - upgrading google-cloud-bigquery"; \
66-
$(PIP) install 'google-cloud-bigquery>=3.0.0' --upgrade; \
66+
$(PIP) install 'google-cloud-bigquery>=3.0.0' \
67+
'pyOpenSSL>=24.0.0' --upgrade; \
6768
fi; \
6869
mv pyproject.toml.backup pyproject.toml; \
6970
echo "Restored original pyproject.toml"

‎docs/concepts/audits.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,38 @@ AUDIT (name price_is_not_null);
158158
SELECT * FROM @this_model
159159
WHERE price IS NULL;
160160
```
161+
### Standalone audits
161162

163+
Standalone audits are defined independently rather than being attached to a specific model. They specify the models they depend on using the `depends_on` property.
164+
165+
Unlike model-level audits, standalone audits can be used to validate data across one or more models without being associated with a single model.
166+
167+
Standalone audits run as scheduled nodes during both `sqlmesh plan` and `sqlmesh run`.
168+
169+
```sql linenums="1"
170+
AUDIT (
171+
name assert_item_price_is_not_null,
172+
dialect spark,
173+
standalone TRUE,
174+
depends_on (
175+
sushi.items
176+
)
177+
);
178+
179+
SELECT *
180+
FROM sushi.items
181+
WHERE
182+
ds BETWEEN @start_ds AND @end_ds
183+
AND price IS NULL;
184+
```
185+
186+
In this example, the audit checks that the `price` column in `sushi.items` does not contain `NULL` values for the selected date range.
187+
188+
Standalone audits can declare dependencies using the `depends_on` property. SQLMesh can often infer dependencies directly from the audit query, but using `depends_on` is recommended when inference isn't sufficient.
189+
190+
!!! note
191+
192+
Standalone audits are non-blocking only. Because they are not associated with a single model, SQLMesh cannot determine which model should be blocked if the audit fails.
162193
## Built-in audits
163194
SQLMesh comes with a suite of built-in generic audits that cover a broad set of common use cases. Built-in audits are blocking by default, but they all have non-blocking counterparts which you can use by appending `_non_blocking` - see [Non-blocking audits](#non-blocking-audits).
164195

‎docs/concepts/models/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ This table lists each engine's support for `TABLE` and `VIEW` object comments:
184184
| DuckDB <=0.9 | N | N |
185185
| DuckDB >=0.10 | Y | Y |
186186
| MySQL | Y | Y |
187-
| MSSQL | N | N |
187+
| MSSQL | Y | Y |
188188
| Postgres | Y | Y |
189189
| GCP Postgres | Y | Y |
190190
| Redshift | Y | N |

‎docs/concepts/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,4 +68,4 @@ SQLMesh automatically runs audits when you apply a `plan` to an environment, or
6868
## Infrastructure and orchestration
6969
Every company's data infrastructure is different. SQLMesh is flexible with regard to which engines and orchestration frameworks you use &mdash; its only requirement is access to the target SQL/analytics engine.
7070

71-
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. SQLMesh it automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.
71+
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. It automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.

‎docs/concepts/plans.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,12 @@ For context, every model has a start date. The start can be specified in [the mo
109109

110110
Because the prod environment supports business operations, prod plans ensure every model is backfilled from its start date until the most recent completed time interval. Due to that restriction, the `plan` command's `--start` and `--end` options are not supported for regular plans against prod. The options are supported for [restatement plans](#restatement-plans) against prod to allow re-processing a subset of existing data.
111111

112+
!!! note "Explicit execution time"
113+
114+
"The most recent completed time interval" is measured relative to the plan's *execution time*, which defaults to now. If you pass an explicit `--execution-time` that is later than the intervals already loaded in prod, the plan extends its end date up to that time and backfills the intervals in between.
115+
116+
For example, if a daily model in prod is loaded through 2025-12-25, running `sqlmesh plan --execution-time '2025-12-28'` backfills the missing 2025-12-26 and 2025-12-27 intervals.
117+
112118
Non-prod plans are typically used for development, so their models can optionally be backfilled for any date range with the `--start` and `--end` options. Limiting the date range makes backfills faster and development more efficient, especially for incremental models using large tables.
113119

114120
#### Model kind limitations

‎docs/guides/configuration.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,16 @@ The examples specify a Snowflake connection whose password is stored in an envir
170170
account: <account>
171171
```
172172

173+
!!! tip "Base64-encoded secrets"
174+
175+
If a secret is distributed base64-encoded in a single environment variable (for example a BigQuery service-account key), pipe the variable through the built-in `b64decode` filter to decode it to text inline:
176+
177+
```yaml
178+
keyfile_json: {{ env_var('BIGQUERY_KEY_B64') | b64decode }}
179+
```
180+
181+
A matching `b64encode` filter is also available. Both return UTF-8 text, so they are intended for string/JSON secrets rather than arbitrary binary data.
182+
173183
=== "Python"
174184

175185
Python accesses environment variables via the `os` library's `environ` dictionary.

‎docs/guides/linter.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ Here are all of SQLMesh's built-in linting rules:
7474
| `invalidselectstarexpansion` | Correctness | The query's top-level selection may be `SELECT *`, but only if SQLMesh can expand the `SELECT *` into individual columns |
7575
| `noselectstar` | Stylistic | The query's top-level selection may not be `SELECT *`, even if SQLMesh can expand the `SELECT *` into individual columns |
7676
| `nomissingaudits` | Governance | SQLMesh did not find any `audits` in the model's configuration to test data quality. |
77+
| `nomissingunittest` | Governance | SQLMesh did not find any `unit tests` associated with the model to test |
7778

7879
### User-defined rules
7980

@@ -126,6 +127,14 @@ Error: Linter detected errors in the code. Please fix them before proceeding.
126127

127128
Use `sqlmesh lint --help` for more information.
128129

130+
You can pass `--local` to run lint without loading state from the configured state connection:
131+
132+
``` bash
133+
$ sqlmesh lint --local
134+
```
135+
136+
This can make linting faster in repositories where all referenced models are loaded from local files. In multi-repository setups, or when linting only a subset of projects, `--local` may cause additional linting errors because SQLMesh will not resolve references or schemas from models that exist only in remote state.
137+
129138

130139
## Applying linting rules
131140

@@ -258,4 +267,4 @@ You may specify that a rule's violation should not error and only log a warning
258267
)
259268
```
260269

261-
SQLMesh will raise an error if the same rule is included in more than one of the `rules`, `warn_rules`, and `ignored_rules` keys since they should be mutually exclusive.
270+
SQLMesh will raise an error if the same rule is included in more than one of the `rules`, `warn_rules`, and `ignored_rules` keys since they should be mutually exclusive.

‎docs/integrations/dlt.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,12 @@ This will create the configuration file and directories, which are found in all
2828

2929
SQLMesh will also automatically generate models to ingest data from the pipeline incrementally. Incremental loading is ideal for large datasets where recomputing entire tables is resource-intensive. In this case utilizing the [`INCREMENTAL_BY_TIME_RANGE` model kind](../concepts/models/model_kinds.md#incremental_by_time_range). However, these model definitions can be customized to meet your specific project needs.
3030

31-
#### Specify the path to the pipelines directory
31+
#### Specify the path to the pipelines working directory
3232

33-
The default location for dlt pipelines is `~/.dlt/pipelines/<pipeline_name>`. If your pipelines are in a [different directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify the path explicitly:
33+
The default location for dlt pipeline working state is `~/.dlt/pipelines/<pipeline_name>`. If dlt stores your pipeline state in a [different pipelines working directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify that directory explicitly. This should be the directory where dlt stores pipeline state, not the directory containing your pipeline scripts:
3434

3535
```bash
36-
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-directory> dialect
36+
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-working-directory> dialect
3737
```
3838

3939
### Generating models on demand
@@ -58,10 +58,10 @@ sqlmesh dlt_refresh <pipeline-name> --force
5858
sqlmesh dlt_refresh <pipeline-name> --table <dlt-table>
5959
```
6060

61-
- **Provide the explicit path to the pipelines directory** (using `--dlt-path`):
61+
- **Provide the explicit path to the pipelines working directory** (using `--dlt-path`):
6262

6363
```bash
64-
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-directory>
64+
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-working-directory>
6565
```
6666

6767
#### Configuration

‎docs/integrations/engines/azuresql.md‎

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,41 @@ pip install "sqlmesh[azuresql]"
1414
```
1515
pip install "sqlmesh[azuresql-odbc]"
1616
```
17+
Set `driver: "pyodbc"` in your connection options.
18+
19+
20+
#### Python Driver (Official Microsoft driver for Azure SQL):
21+
See [`mssql-python`](https://pypi.org/project/mssql-python/) for more information.
22+
23+
```
24+
pip install "sqlmesh[azuresql-mssql-python]"
25+
```
26+
27+
Set `driver: "mssql-python"` in your connection options. This driver supports
28+
[Entra ID auth](https://github.com/microsoft/mssql-python/wiki/Microsoft-Entra-ID-support),
29+
for detailed connection options see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database).
30+
31+
!!! note
32+
The `mssql-python` driver [requires](https://pypi.org/project/mssql-python/) `python >= 3.10`.
33+
1734

1835
### Connection options
1936

20-
| Option | Description | Type | Required |
21-
| ----------------- | ---------------------------------------------------------------- | :----------: | :------: |
22-
| `type` | Engine type name - must be `azuresql` | string | Y |
23-
| `host` | The hostname of the Azure SQL server | string | Y |
24-
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
25-
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
26-
| `port` | The port number of the Azure SQL server | int | N |
27-
| `database` | The target database | string | N |
28-
| `charset` | The character set used for the connection | string | N |
29-
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
30-
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
31-
| `appname` | The application name to use for the connection | string | N |
32-
| `conn_properties` | The list of connection properties | list[string] | N |
33-
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
34-
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
35-
| `driver_name` | The driver name to use for the connection. E.g., *ODBC Driver 18 for SQL Server* | string | N |
36-
| `odbc_properties` | The dict of ODBC connection properties. E.g., authentication: ActiveDirectoryServicePrincipal. See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |
37+
| Option | Description | Type | Required |
38+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
39+
| `type` | Engine type name - must be `azuresql` | string | Y |
40+
| `host` | The hostname of the Azure SQL server | string | Y |
41+
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
42+
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
43+
| `port` | The port number of the Azure SQL server | int | N |
44+
| `database` | The target database | string | N |
45+
| `charset` | The character set used for the connection | string | N |
46+
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
47+
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
48+
| `login_attempts` | The number of reconnection attempts before failing. Default: 1 <br><br>*This option only applies to the `mssql-python` driver. | int | N |
49+
| `appname` | The application name to use for the connection | string | N |
50+
| `conn_properties` | The list of connection properties | list[string] | N |
51+
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
52+
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
53+
| `driver_name` | The driver name to use for the connection (e.g., *ODBC Driver 18 for SQL Server*). | string | N |
54+
| `odbc_properties` | The dict of ODBC connection properties (e.g., *authentication: ActiveDirectoryServicePrincipal*). See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16).<br><br>*For the `mssql-python` driver, please see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database). | dict | N |

0 commit comments

Comments
 (0)