You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 559f1d3
Browse filesBrowse the repository at this point in the historyBrowse files
Fix inline sequence rendering after IN (#629) (#1013)
* Fix inline sequence rendering after IN (#629)
Since 4.0.4, inline parameters render every Python sequence as an ARRAY
literal, so the documented `x IN %(ids)s` pattern produces
`x IN ARRAY(1,2,3)`, which Spark rejects with PARSE_SYNTAX_ERROR.
A sequence whose marker directly follows IN (or NOT IN) now renders as a
parenthesised list again. Everywhere else, including `IN (%(p)s)` where
the caller wrote the parentheses, it stays an ARRAY literal, so inline
writes to ARRAY columns keep working. Because `x IN ARRAY(...)` never
parses, no query that runs today renders differently.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: cathleen.yan <58714163+cathleeny@users.noreply.github.com>
* Choose the IN-list rendering per marker and probe only sequences
Follow-ups from review of the inline IN fix:
- Decide per marker instead of per parameter name. A name used both
directly after IN and elsewhere now keeps its ARRAY rendering at the
other markers. Before, every occurrence became `(...)`, which could
break a query that runs on main, e.g. one with a commented-out filter:
`arr = %(tags)s -- AND tag IN %(tags)s`.
- Probe only sequence values and give every other marker its real
escaped value, so a `%d` or `%(n)d` marker elsewhere in the query no
longer turns the fix off. Probe tokens carry a random nonce so that a
string value cannot pass for one.
- Leave escape_args exactly as it is on main and move the logic into
ParamEscaper.render_inline, which Cursor._prepare_inline_parameters
now calls.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Render tuples in an IN list as rows; skip probing without sequences
Follow-ups from review:
- Inside an IN list a tuple now renders as a row, so a multi-column
`(a, b) IN %(pairs)s` with `[(1, 2), (3, 4)]` renders as
`(a, b) IN ((1,2),(3,4))`, as it did in 3.x. Lists still render as
ARRAY literals so ARRAY columns can be matched. Spark does not widen
the field types of a row, so each value must have its column's type;
docs/parameters.md says so.
- When no parameter is a sequence, render_inline skips the probe pass
and renders exactly as escape_args and inject_parameters do on main.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Ignore an IN that ends a line comment; keep values out of probing
Review bot follow-up: a `--` comment line that ends with the word IN
made the marker on the next line render as a list. For example
WHERE arr = -- tags we filter IN
%(tags)s
rendered as `arr = ('a','b')` instead of `arr = ARRAY('a','b')`, which
breaks a query that runs on main. A match is now ignored when a `--`
comes before its IN on the same line and the marker is on a later line.
The probe pass also replaces escaped strings with '' (numbers stay, so
%d still formats), so a value containing `--` cannot hide a real IN.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Accept a set directly after IN in inline parameters
Up to 4.0.3 the inline escaper rendered any iterable as `(...)`, so a
set after IN worked. 4.0.4 switched to Sequence for ARRAY rendering and
sets started raising ProgrammingError.
A set, frozenset or dict key view is accepted again when every marker
it fills directly follows IN. Its items are sorted so the SQL text is
stable, and a set of tuples renders as rows. Anywhere else a set raises
the same ProgrammingError as before, since a set has no order to keep
in an ARRAY.
Also adds a CHANGELOG entry for the inline IN fix.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Describe IN detection as a text heuristic
Review bot follow-up: the comments and docs described comment handling
as if it were settled. They now say that IN is found by matching the
SQL text rather than parsing it, which case is handled (an IN that ends
a `--` comment line) and which is not (an IN inside a string literal or
block comment directly before a marker).
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Don't promise identical SQL text for sets
Review bot follow-up: sorting a set's items does not give a stable
order when items compare inconsistently, such as float("nan") among
other numbers. The SQL still matches the same rows, but the docstring
and docs/parameters.md no longer say the query text is always the same.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Keep bytes after IN rendering as an ARRAY, as on main
Review bot follow-up: bytes and bytearray are sequences of ints, so
`x IN %(v)s` with b"ab" rendered as `x IN (97,98)`. That is valid SQL
and silently compares x with the byte values, where main renders
`x IN ARRAY(97,98)` and Spark rejects it. bytes and bytearray now stay
out of the IN-list rendering, so they keep main's behaviour.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
* Rewrap a long docstring; make the probe-token test guard the nonce
Review bot follow-ups:
- Rewrap the _markers_following_in docstring, where an edit had joined
two lines into one of about 140 characters.
- test_probe_shaped_string_value_is_not_a_probe could not fail without
the nonce check, because string values are blanked during the probe
pass. The test now also puts probe-shaped text in the operation
itself, which reaches the probe pass unchanged, so only the nonce
check keeps it from counting.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
---------
Signed-off-by: cathleen.yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,7 @@
1
1
# Release History
2
2
3
3
# Unreleased
4
+
- Fix inline parameters (`use_inline_params=True`) so a sequence directly after `IN` renders as a list again: `x IN %(ids)s` renders as `x IN (1,2,3)` instead of `x IN ARRAY(1,2,3)`, which Spark rejects. Inside the list, tuples render as rows for a multi-column `IN`, and a `set` or `frozenset` is accepted again. Sequences anywhere else still render as `ARRAY(...)` ([#629](https://github.com/databricks/databricks-sql-python/issues/629))
4
5
5
6
# 4.9.0 (2026-11-04)
6
7
- Upgrade Databricks SQL Kernel to 1.4.0, which keeps Lakehouse Real-Time session affinity and flushes shared telemetry only after the final session closes.
Copy file name to clipboardExpand all lines: docs/parameters.md
+19Lines changed: 19 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -235,6 +235,25 @@ Output:
235
235
SELECT field FROM table WHERE field IN (1,2,3,4,5)
236
236
```
237
237
238
+
The marker must directly follow `IN` (or `NOT IN`), as in `IN %(value_list)s`. Anywhere else, including inside parentheses you write yourself such as `IN (%(value_list)s)`, a sequence is rendered as an `ARRAY(...)` literal.
239
+
240
+
The connector finds `IN` by reading the query text, not by parsing SQL. It ignores an `IN` at the end of a `--` comment line, but an `IN` inside a string literal, directly before a marker, is still treated as the keyword.
241
+
242
+
Inside the list, a tuple is rendered as a row and a list as an `ARRAY(...)` literal. For a multi-column `IN`, pass tuples:
243
+
244
+
```python
245
+
query ="SELECT * FROM table WHERE (id, tag) IN %(pairs)s"
246
+
parameters = {"pairs": [(1, "a"), (2, "b")]}
247
+
```
248
+
249
+
```sql
250
+
SELECT*FROM table WHERE (id, tag) IN ((1,'a'),(2,'b'))
251
+
```
252
+
253
+
Spark does not convert the types of the values inside a row, so each value must have the same type as its column. For example, the literal `1` is an `INT` and does not match a `BIGINT` column.
254
+
255
+
A `set` or `frozenset` can also be passed directly after `IN`. Its items are sorted, which usually keeps the query text the same from run to run. Items without a consistent order, such as `float("nan")`, can still change position. Anywhere else a set raises `ProgrammingError`, because a set has no order to keep in an `ARRAY(...)` literal.
256
+
238
257
**Note**: this behavior is not specified by PEP-249 and only works due to the specific implementation of our inline renderer.
0 commit comments