Skip to content

Commit 559f1d3

Browse files
cathleenyIsaac
andauthored
Fix inline sequence rendering after IN (#629) (#1013)
* Fix inline sequence rendering after IN (#629) Since 4.0.4, inline parameters render every Python sequence as an ARRAY literal, so the documented `x IN %(ids)s` pattern produces `x IN ARRAY(1,2,3)`, which Spark rejects with PARSE_SYNTAX_ERROR. A sequence whose marker directly follows IN (or NOT IN) now renders as a parenthesised list again. Everywhere else, including `IN (%(p)s)` where the caller wrote the parentheses, it stays an ARRAY literal, so inline writes to ARRAY columns keep working. Because `x IN ARRAY(...)` never parses, no query that runs today renders differently. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: cathleen.yan <58714163+cathleeny@users.noreply.github.com> * Choose the IN-list rendering per marker and probe only sequences Follow-ups from review of the inline IN fix: - Decide per marker instead of per parameter name. A name used both directly after IN and elsewhere now keeps its ARRAY rendering at the other markers. Before, every occurrence became `(...)`, which could break a query that runs on main, e.g. one with a commented-out filter: `arr = %(tags)s -- AND tag IN %(tags)s`. - Probe only sequence values and give every other marker its real escaped value, so a `%d` or `%(n)d` marker elsewhere in the query no longer turns the fix off. Probe tokens carry a random nonce so that a string value cannot pass for one. - Leave escape_args exactly as it is on main and move the logic into ParamEscaper.render_inline, which Cursor._prepare_inline_parameters now calls. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Render tuples in an IN list as rows; skip probing without sequences Follow-ups from review: - Inside an IN list a tuple now renders as a row, so a multi-column `(a, b) IN %(pairs)s` with `[(1, 2), (3, 4)]` renders as `(a, b) IN ((1,2),(3,4))`, as it did in 3.x. Lists still render as ARRAY literals so ARRAY columns can be matched. Spark does not widen the field types of a row, so each value must have its column's type; docs/parameters.md says so. - When no parameter is a sequence, render_inline skips the probe pass and renders exactly as escape_args and inject_parameters do on main. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Ignore an IN that ends a line comment; keep values out of probing Review bot follow-up: a `--` comment line that ends with the word IN made the marker on the next line render as a list. For example WHERE arr = -- tags we filter IN %(tags)s rendered as `arr = ('a','b')` instead of `arr = ARRAY('a','b')`, which breaks a query that runs on main. A match is now ignored when a `--` comes before its IN on the same line and the marker is on a later line. The probe pass also replaces escaped strings with '' (numbers stay, so %d still formats), so a value containing `--` cannot hide a real IN. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Accept a set directly after IN in inline parameters Up to 4.0.3 the inline escaper rendered any iterable as `(...)`, so a set after IN worked. 4.0.4 switched to Sequence for ARRAY rendering and sets started raising ProgrammingError. A set, frozenset or dict key view is accepted again when every marker it fills directly follows IN. Its items are sorted so the SQL text is stable, and a set of tuples renders as rows. Anywhere else a set raises the same ProgrammingError as before, since a set has no order to keep in an ARRAY. Also adds a CHANGELOG entry for the inline IN fix. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Describe IN detection as a text heuristic Review bot follow-up: the comments and docs described comment handling as if it were settled. They now say that IN is found by matching the SQL text rather than parsing it, which case is handled (an IN that ends a `--` comment line) and which is not (an IN inside a string literal or block comment directly before a marker). Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Don't promise identical SQL text for sets Review bot follow-up: sorting a set's items does not give a stable order when items compare inconsistently, such as float("nan") among other numbers. The SQL still matches the same rows, but the docstring and docs/parameters.md no longer say the query text is always the same. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Keep bytes after IN rendering as an ARRAY, as on main Review bot follow-up: bytes and bytearray are sequences of ints, so `x IN %(v)s` with b"ab" rendered as `x IN (97,98)`. That is valid SQL and silently compares x with the byte values, where main renders `x IN ARRAY(97,98)` and Spark rejects it. bytes and bytearray now stay out of the IN-list rendering, so they keep main's behaviour. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> * Rewrap a long docstring; make the probe-token test guard the nonce Review bot follow-ups: - Rewrap the _markers_following_in docstring, where an edit had joined two lines into one of about 140 characters. - test_probe_shaped_string_value_is_not_a_probe could not fail without the nonce check, because string values are blanked during the probe pass. The test now also puts probe-shaped text in the operation itself, which reaches the probe pass unchanged, so only the nonce check keeps it from counting. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> --------- Signed-off-by: cathleen.yan <58714163+cathleeny@users.noreply.github.com> Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> Co-authored-by: Isaac <no-reply@databricks.com>
1 parent bfd1f24 commit 559f1d3

7 files changed

Lines changed: 785 additions & 5 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Release History
22

33
# Unreleased
4+
- Fix inline parameters (`use_inline_params=True`) so a sequence directly after `IN` renders as a list again: `x IN %(ids)s` renders as `x IN (1,2,3)` instead of `x IN ARRAY(1,2,3)`, which Spark rejects. Inside the list, tuples render as rows for a multi-column `IN`, and a `set` or `frozenset` is accepted again. Sequences anywhere else still render as `ARRAY(...)` ([#629](https://github.com/databricks/databricks-sql-python/issues/629))
45

56
# 4.9.0 (2026-11-04)
67
- Upgrade Databricks SQL Kernel to 1.4.0, which keeps Lakehouse Real-Time session affinity and flushes shared telemetry only after the final session closes.

‎docs/parameters.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,25 @@ Output:
235235
SELECT field FROM table WHERE field IN (1,2,3,4,5)
236236
```
237237

238+
The marker must directly follow `IN` (or `NOT IN`), as in `IN %(value_list)s`. Anywhere else, including inside parentheses you write yourself such as `IN (%(value_list)s)`, a sequence is rendered as an `ARRAY(...)` literal.
239+
240+
The connector finds `IN` by reading the query text, not by parsing SQL. It ignores an `IN` at the end of a `--` comment line, but an `IN` inside a string literal, directly before a marker, is still treated as the keyword.
241+
242+
Inside the list, a tuple is rendered as a row and a list as an `ARRAY(...)` literal. For a multi-column `IN`, pass tuples:
243+
244+
```python
245+
query = "SELECT * FROM table WHERE (id, tag) IN %(pairs)s"
246+
parameters = {"pairs": [(1, "a"), (2, "b")]}
247+
```
248+
249+
```sql
250+
SELECT * FROM table WHERE (id, tag) IN ((1,'a'),(2,'b'))
251+
```
252+
253+
Spark does not convert the types of the values inside a row, so each value must have the same type as its column. For example, the literal `1` is an `INT` and does not match a `BIGINT` column.
254+
255+
A `set` or `frozenset` can also be passed directly after `IN`. Its items are sorted, which usually keeps the query text the same from run to run. Items without a consistent order, such as `float("nan")`, can still change position. Anywhere else a set raises `ProgrammingError`, because a set has no order to keep in an `ARRAY(...)` literal.
256+
238257
**Note**: this behavior is not specified by PEP-249 and only works due to the specific implementation of our inline renderer.
239258

240259
### Migrating to native parameters

‎src/databricks/sql/client.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@
4646
from databricks.sql.backend.databricks_client import DatabricksClient
4747
from databricks.sql.utils import (
4848
ParamEscaper,
49-
inject_parameters,
5049
transform_paramstyle,
5150
ColumnTable,
5251
ColumnQueue,
@@ -1211,8 +1210,7 @@ def _prepare_inline_parameters(
12111210
The list is always empty because native parameters are never used under the inline approach
12121211
"""
12131212

1214-
escaped_values = self.escaper.escape_args(params)
1215-
rendered_statement = inject_parameters(stmt, escaped_values)
1213+
rendered_statement = self.escaper.render_inline(stmt, params)
12161214

12171215
return rendered_statement, NO_NATIVE_PARAMS
12181216

‎src/databricks/sql/utils.py‎

Lines changed: 277 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,27 @@
11
from __future__ import annotations
2-
from typing import Any, Dict, List, Optional, Tuple, Union, Sequence, TYPE_CHECKING
2+
from typing import (
3+
Any,
4+
Callable,
5+
Dict,
6+
List,
7+
Optional,
8+
Set,
9+
Tuple,
10+
Union,
11+
Sequence,
12+
TYPE_CHECKING,
13+
)
314

415
from dateutil import parser
516
import datetime
617
import decimal
718
from abc import ABC, abstractmethod
819
from collections import OrderedDict, namedtuple
9-
from collections.abc import Mapping
20+
from collections.abc import Mapping, Set as AbstractSet
1021
from decimal import Decimal
1122
from enum import Enum
1223
import re
24+
import secrets
1325

1426
import lz4.frame
1527

@@ -557,6 +569,160 @@ def user_friendly_error_message(self, no_retry_reason, attempt, elapsed):
557569
return user_friendly_error_message + ". " + error_context
558570

559571

572+
# A probe token directly after the IN keyword (also matches NOT IN). Where Spark's grammar accepts a
573+
# value directly after IN, in the IN predicate and in PIVOT, it requires a parenthesised list, so
574+
# `IN ARRAY(...)` is a syntax error. (The one other place, `position(a IN b)`, takes a string and
575+
# rejects an ARRAY too.) Requiring the marker to follow IN directly leaves `IN (%(p)s)` alone: there
576+
# the caller's parentheses already form the list and an ARRAY value is a valid list element.
577+
#
578+
# This matches the SQL text rather than parsing it, so it is a heuristic. It skips an IN that ends
579+
# a `--` comment line (see `_in_ends_line_comment`), but an IN inside a string literal or a block
580+
# comment can still count as the keyword when a marker follows it.
581+
_IN_MARKER_PROBE = re.compile(r"\bIN\s*\x00([0-9a-f]+):(\d+)\x00", re.IGNORECASE)
582+
583+
584+
def _in_ends_line_comment(text: str, match: re.Match[str]) -> bool:
585+
"""Whether the IN that `match` found looks like it is in a `--` comment, with the marker on a
586+
later line.
587+
588+
This is a check on the text, not a parse. In that case the marker is most likely live code, so
589+
it keeps its ARRAY rendering. A marker on the same line as a commented IN is usually inside the
590+
comment too, so it is not checked. A `--` inside a string literal earlier on the line also
591+
counts, which falls back to the ARRAY rendering.
592+
"""
593+
if not re.search(r"[\r\n]", match.group(0)):
594+
return False
595+
line_start = max(
596+
text.rfind("\n", 0, match.start()), text.rfind("\r", 0, match.start())
597+
)
598+
return "--" in text[line_start + 1 : match.start()]
599+
600+
601+
def _renders_as_array(value: Any) -> bool:
602+
"""Whether ParamEscaper.escape_item renders `value` as an ARRAY literal."""
603+
return isinstance(value, Sequence) and not isinstance(value, str)
604+
605+
606+
def _is_set(value: Any) -> bool:
607+
"""Whether `value` is a set, frozenset or dict key view. A set has no order, so it can only
608+
render as an IN list, never as an ARRAY."""
609+
return isinstance(value, AbstractSet)
610+
611+
612+
def _renders_as_in_list(value: Any) -> bool:
613+
"""Whether `value` renders as an IN list when its marker directly follows IN.
614+
615+
bytes and bytearray are sequences of ints, so escape_item renders them as ARRAY literals of
616+
byte values. They are left out here, so `x IN %s` with bytes still fails loudly as it does on
617+
main, instead of quietly comparing x with the byte values.
618+
"""
619+
if isinstance(value, (bytes, bytearray)):
620+
return False
621+
return _renders_as_array(value) or _is_set(value)
622+
623+
624+
def _in_list_items(value: Any) -> Any:
625+
"""Return the items to render for an IN list, with a set's items sorted.
626+
627+
Sorting usually makes the SQL text the same from run to run, but not always: items without a
628+
consistent order, such as `float("nan")` among other numbers, can still come out in a
629+
different order. The IN list matches the same rows either way.
630+
"""
631+
if not _is_set(value):
632+
return value
633+
try:
634+
return sorted(value)
635+
except TypeError:
636+
# The items do not compare with each other, for example None and an int.
637+
return sorted(value, key=repr)
638+
639+
640+
class _InListOnly:
641+
"""Stands in for an escaped set, which can only be rendered as an IN list.
642+
643+
`render_inline` rejects a set at any other marker before formatting. Raising here as well
644+
keeps an unescaped set from ever reaching the SQL text.
645+
"""
646+
647+
def __init__(self, value: Any) -> None:
648+
self.value = value
649+
650+
def __str__(self) -> str:
651+
raise ProgrammingError("Unsupported object {}".format(self.value))
652+
653+
__repr__ = __str__
654+
655+
656+
class _MarkerProbe:
657+
"""Formats as a token that identifies the marker it was substituted into."""
658+
659+
def __init__(self, nonce: str, index: int) -> None:
660+
self._token = f"\x00{nonce}:{index}\x00"
661+
662+
def __str__(self) -> str:
663+
return self._token
664+
665+
__repr__ = __str__
666+
667+
668+
class _NumberedLookups:
669+
"""Mapping stand-in for %-formatting that numbers its lookups.
670+
671+
%-formatting looks up each %(name)s marker once, left to right, so lookup n is always the n-th
672+
named marker of the operation. `value_for(n, name)` supplies the value substituted there.
673+
"""
674+
675+
def __init__(
676+
self, value_for: Callable[[int, str], Any], escaped: Dict[str, Any]
677+
) -> None:
678+
self._value_for = value_for
679+
self._escaped = escaped
680+
self._count = 0
681+
682+
def __getitem__(self, name: str) -> Any:
683+
index = self._count
684+
self._count += 1
685+
return self._value_for(index, name)
686+
687+
def __str__(self) -> str:
688+
# A bare %s marker formats the whole mapping; keep rendering the escaped dict there.
689+
return str(self._escaped)
690+
691+
__repr__ = __str__
692+
693+
694+
def _reject_sets_outside_in_lists(
695+
parameters: Any, in_list_markers: Set[int], marker_keys: Dict[int, Any]
696+
) -> None:
697+
"""Raise the error `escape_args` raises for a set, unless the set is used and every marker
698+
it fills renders as an IN list."""
699+
items = (
700+
parameters.items() if isinstance(parameters, dict) else enumerate(parameters)
701+
)
702+
for key, value in items:
703+
if not _is_set(value):
704+
continue
705+
markers = {index for index, k in marker_keys.items() if k == key}
706+
if not markers or not markers <= in_list_markers:
707+
raise ProgrammingError("Unsupported object {}".format(value))
708+
709+
710+
def _format_markers(
711+
operation: str, escaped: Any, value_for: Callable[[int, Any], Any]
712+
) -> str:
713+
"""%-format `operation`, taking the value of marker n from `value_for(n, key)`.
714+
715+
`escaped` holds the escaped parameter values. `key` is the marker's name when it is a dict and
716+
the marker's position otherwise.
717+
"""
718+
values: Any
719+
if isinstance(escaped, dict):
720+
values = _NumberedLookups(value_for, escaped)
721+
else:
722+
values = tuple(value_for(i, i) for i in range(len(escaped)))
723+
return inject_parameters(operation, values)
724+
725+
560726
# Taken from PyHive
561727
class ParamEscaper:
562728
_DATE_FORMAT = "%Y-%m-%d"
@@ -571,6 +737,115 @@ def escape_args(self, parameters):
571737
else:
572738
raise ProgrammingError("Unsupported param format: {}".format(parameters))
573739

740+
def render_inline(self, operation: str, parameters) -> str:
741+
"""Render `parameters` into the markers of `operation` as inline SQL literals.
742+
743+
Values are escaped as `escape_args` escapes them, with one exception: a sequence at a
744+
marker that directly follows IN (e.g. `x IN %(ids)s`) renders as the parenthesised list
745+
Spark requires there, `x IN (1,2,3)`, instead of `x IN ARRAY(1,2,3)`, which Spark rejects.
746+
The choice is made per marker, so a parameter used both after IN and elsewhere keeps its
747+
ARRAY rendering at the other markers. Inside the list, tuples render as rows and lists as
748+
ARRAY literals.
749+
750+
IN is found by matching the SQL text, not by parsing it (see `_IN_MARKER_PROBE`), so an
751+
unusual query, such as one with IN inside a string literal just before a marker, can
752+
still get a list where an ARRAY was meant.
753+
754+
A set (or frozenset, or dict key view) is accepted only where every marker it fills
755+
directly follows IN, and its items are sorted. Anywhere else it raises the same
756+
`ProgrammingError` as `escape_args`, because a set has no order to put into an ARRAY.
757+
"""
758+
escaped = self._escape_values(parameters)
759+
values = parameters.values() if isinstance(parameters, dict) else parameters
760+
if not any(_renders_as_in_list(value) for value in values):
761+
# Only a sequence or a set can render as an IN list, so without one render exactly
762+
# as before.
763+
return inject_parameters(operation, escaped)
764+
765+
in_list_markers, marker_keys = self._markers_following_in(
766+
operation, parameters, escaped
767+
)
768+
_reject_sets_outside_in_lists(parameters, in_list_markers, marker_keys)
769+
770+
def render_marker(index: int, key: Any) -> Any:
771+
if index in in_list_markers:
772+
return self.escape_in_list(_in_list_items(parameters[key]))
773+
return escaped[key]
774+
775+
return _format_markers(operation, escaped, render_marker)
776+
777+
def _escape_values(self, parameters):
778+
"""Escape like `escape_args`, except that a set becomes an `_InListOnly` placeholder
779+
instead of raising, so `render_inline` can decide where it is allowed."""
780+
781+
def escape(value: Any) -> Any:
782+
return _InListOnly(value) if _is_set(value) else self.escape_item(value)
783+
784+
if isinstance(parameters, dict):
785+
return {k: escape(v) for k, v in parameters.items()}
786+
elif isinstance(parameters, (list, tuple)):
787+
return tuple(escape(x) for x in parameters)
788+
else:
789+
raise ProgrammingError("Unsupported param format: {}".format(parameters))
790+
791+
def _markers_following_in(
792+
self, operation: str, parameters, escaped
793+
) -> Tuple[Set[int], Dict[int, Any]]:
794+
"""Return the numbers of the markers that directly follow IN and receive a value that can
795+
render as an IN list, and the parameter key that each marker number refers to.
796+
797+
Markers are numbered as `_format_markers` numbers them. To find them, the operation is
798+
formatted once through the same %-formatting as the real substitution, so `%%` and
799+
positional markers line up exactly. Each value that can render as an IN list (see
800+
`_renders_as_in_list`) is replaced by a probe token. Numbers keep their value, so
801+
markers such as `%d` still format, and escaped strings are replaced by an empty string
802+
literal, so the text of a value cannot change the result. If the operation does not
803+
format, both results are empty.
804+
"""
805+
# The nonce stops text that only looks like a probe token from being taken for one.
806+
nonce = secrets.token_hex(8)
807+
marker_keys: Dict[int, Any] = {}
808+
809+
def probe_marker(index: int, key: Any) -> Any:
810+
# A missing name raises KeyError here, as it does when rendering.
811+
escaped_value = escaped[key]
812+
marker_keys[index] = key
813+
value = parameters[key]
814+
if _renders_as_in_list(value):
815+
return _MarkerProbe(nonce, index)
816+
# Replace text with '' so the content of a value cannot change the result. A string
817+
# of two or more characters formats under exactly the markers '' does. A one-character
818+
# string can only be a single-digit Decimal, which %c accepts, so it is kept.
819+
if isinstance(escaped_value, str) and len(escaped_value) > 1:
820+
return "''"
821+
return escaped_value
822+
823+
try:
824+
probed_operation = _format_markers(operation, escaped, probe_marker)
825+
except (KeyError, TypeError, ValueError):
826+
# Rendering raises the same error: a probe only replaces a sequence, and a marker that
827+
# cannot format a probe (such as %d) cannot format the sequence's ARRAY string either.
828+
# A set gets the ProgrammingError escape_args raises, since none of its markers count.
829+
return set(), {}
830+
831+
in_list_markers = {
832+
int(match.group(2))
833+
for match in _IN_MARKER_PROBE.finditer(probed_operation)
834+
if match.group(1) == nonce
835+
and not _in_ends_line_comment(probed_operation, match)
836+
}
837+
return in_list_markers, marker_keys
838+
839+
def escape_in_list(self, item):
840+
return "(" + ",".join(map(str, map(self._escape_in_list_element, item))) + ")"
841+
842+
def _escape_in_list_element(self, item):
843+
# Inside an IN list a tuple is a row, as in `(a, b) IN ((1, 2), (3, 4))`. A list stays an
844+
# ARRAY literal so ARRAY columns can still be matched, as in `arr IN (ARRAY(1, 2))`.
845+
if isinstance(item, tuple):
846+
return self.escape_in_list(item)
847+
return self.escape_item(item)
848+
574849
def escape_number(self, item):
575850
return item
576851

0 commit comments

Comments
 (0)