Skip to content

Preserve PostgreSQL substring offset semantics - #332

Open
fallintoplace wants to merge 1 commit into
ClickHouse:mainfrom
fallintoplace:fix/substring-pushdown-semantics
Open

Preserve PostgreSQL substring offset semantics#332
fallintoplace wants to merge 1 commit into
ClickHouse:mainfrom
fallintoplace:fix/substring-pushdown-semantics

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

pg_clickhouse translates PostgreSQL substring functions to ClickHouse for remote execution:

PostgreSQL overload ClickHouse function
substring(text, ...) / substr(text, ...) substringUTF8
substring(bytea, ...) / substr(bytea, ...) substring

Those mappings are only semantically safe for a subset of PostgreSQL bounds. This change retains pushdown for safe constant bounds and evaluates every other case locally in PostgreSQL.

PostgreSQL and ClickHouse differ

The difference affects both mappings, not just text values:

Type PostgreSQL expression PostgreSQL result ClickHouse behavior before this change
text substring('val1' FROM -1) val1 A negative start counts from the end
text substring('val1' FROM 0 FOR 2) v Empty result
text substring('val1' FROM 2 FOR -1) Error: negative substring length not allowed Returns a shortened value
bytea substring('val1'::bytea FROM -1 FOR 3) v A negative start counts from the end
bytea substring('val1'::bytea FROM 0 FOR 2) v Empty result
bytea substring('val1'::bytea FROM 2 FOR -1) Error: negative substring length not allowed Returns a shortened value

Unconditionally shipping these calls changes results and can suppress a PostgreSQL error.

Approach

Push down substring and substr only when their bounds are known-safe constants at planning time:

Function form Pushed down when Evaluated locally when
substring(value, start) / substr(value, start) start is a non-NULL constant >= 1 start is NULL, non-constant, zero, or negative
substring(value, start, length) / substr(value, start, length) start >= 1 and length >= 0, both non-NULL constants Either bound is NULL, non-constant, or outside that range

The guard is applied consistently to text and bytea overloads. Safe cases such as substring(val, 1, 3) and substr(val, 2) continue to use ClickHouse. A parameterized offset is evaluated locally because its value is unavailable when the remote expression is planned.

Tests

Regression coverage includes:

  • text values with negative and zero starts
  • bytea values with a negative start
  • PostgreSQL's negative-length error
  • a generic prepared plan with a parameterized offset
  • EXPLAIN output confirming unsafe calls stay out of the remote SQL

Verified with:

make TESTS=test/sql/functions.sql PGUSER=postgres installcheck

on PostgreSQL 19 against ClickHouse 26.3.17.56 and 23.3.22.3.

Comment thread src/shipable.c Outdated
}

start = (Expr*)lsecond(fn->args);
if (!IsA(start, Const)) {

@serprex serprex Aug 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we push down conditional logic to match postgres instead?

cc @JoshDreamland who recently did this sort of thing to get correct NULL handling with IN in #317

@fallintoplace
fallintoplace force-pushed the fix/substring-pushdown-semantics branch from d86b3f6 to 688c16d Compare August 3, 2026 16:43
@theory theory added pushdown Improvements to query pushdown functions Improve function pushdown labels Aug 3, 2026
@serprex
serprex requested a review from JoshDreamland August 3, 2026 17:05
@fallintoplace
fallintoplace force-pushed the fix/substring-pushdown-semantics branch from 688c16d to 7e77665 Compare August 3, 2026 17:14
ClickHouse interprets non-positive substring starts and negative lengths differently from PostgreSQL. Directly pushing down substring() and substr() can therefore change results or suppress PostgreSQL's negative-length error.

Deparse text and bytea substring calls with PostgreSQL-compatible start and length adjustments, preserving NULL strictness and raising the PostgreSQL error for negative lengths. Keep safe constant bounds on the direct ClickHouse path.

Add regression coverage for text and bytea offsets, negative lengths, NULL arguments, and generic plans.
@fallintoplace
fallintoplace force-pushed the fix/substring-pushdown-semantics branch from 7e77665 to 09b2525 Compare August 5, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Improve function pushdown pushdown Improvements to query pushdown

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants