fix(optimizer): dedupe colliding star expanded aliases by fivetran-kwoodbeck · Pull Request #7872 · tobymao/sqlglot

fivetran-kwoodbeck · 2026-07-14T17:23:53Z

When qualify_columns expands a SELECT * over sources with overlapping column names (e.g. two joined tables that both have an id column), it causes problems:

it produces duplicate output aliases in the projection list,
it can canonicalize into the wrong query,
it can cause different queries to collide on the same output

This is a problem when that expanded scope is itself re-exposed to an outer scope via another table.* or alias.*, since duplicate aliases at that point collapse distinct columns under the same name.

Sample Query (wrong canonicalization)

Schema: {"x": {"id": "INT", "name": "TEXT"}, "y": {"id": "INT", "name": "TEXT"}}
SELECT * FROM x CROSS JOIN (SELECT * FROM y AS a CROSS JOIN x AS b) AS w;

Output (before fix), this is a different query (vs input):

WITH "w" AS (SELECT "a"."id" AS "id", "a"."name" AS "name", "b"."id" AS "id", "b"."name" AS "name" FROM "y" AS "a" CROSS JOIN "x" AS "b") SELECT "x"."id" AS "id", "x"."name" AS "name", "w"."id" AS "id", "w"."name" AS "name", "w"."id" AS "id", "w"."name" AS "name" FROM "x" AS "x" CROSS JOIN "w" AS "w";

Expected output (after fix), same query as input:

SELECT "s"."id" AS "id", "s"."name" AS "name", "s"."id_2" AS "id_2", "s"."name_2" AS "name_2" FROM (SELECT "x"."id" AS "id", "x"."name" AS "name", "y"."id" AS "id_2", "y"."name" AS "name_2" FROM "x" AS "x" CROSS JOIN "y" AS "y") AS "s";

Sample Queries (for query collision):

Schema: {"u": {"v": "INT"}, "w": {"v": "INT"}}
Q_1: SELECT s.v, s.v FROM (SELECT a.v AS v, b.v AS v FROM u AS a CROSS JOIN w AS b) AS s;
Q_2: SELECT * FROM (SELECT * FROM u AS a CROSS JOIN w AS b) AS s;

These are different queries that return different data, but they both collide into:

SELECT "s"."v" AS "v", "s"."v" AS "v" FROM (SELECT "a"."v" AS "v", "b"."v" AS "v" FROM "u" AS "a" CROSS JOIN "w" AS "b") AS "s";

Expected output:

Q_1: SELECT "s"."v" AS "v", "s"."v" AS "v" FROM (SELECT "a"."v" AS "v", "b"."v" AS "v" FROM "u" AS "a" CROSS JOIN "w" AS "b") AS "s";
Q_2: SELECT "s"."v" AS "v", "s"."v_2" AS "v_2" FROM (SELECT "a"."v" AS "v", "b"."v" AS "v_2" FROM "u" AS "a" CROSS JOIN "w" AS "b") AS "s";

georgesittas

@fivetran-kwoodbeck I don't think we should bother dealing with this. These queries seem pathological and the amount of complexity and overhead I'm seeing in this PR makes me feel like this is definitely not worth it.

Postgres' behavior for example demonstrates, imo, why something like this in any normal system / pipeline would be pretty much unusable:

georgesittas=# SELECT * FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOIN t2 AS b) AS w;
 c | c | c
---+---+---
 1 | 1 | 2
(1 row)

georgesittas=# SELECT c FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOIN t2 AS b) AS w;
ERROR:  column reference "c" is ambiguous
LINE 1: SELECT c FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOI...
               ^

github-actions · 2026-07-14T18:25:49Z

SQLGlot Integration Test Results

❌ 5 regressions — see details below

Comparing:

this branch (sqlglot:optimizer/bug-query-star-collision @ sqlglot c4a5e51)
baseline (main @ sqlglot 98e2b3f)

By Dialect

dialect	main	feature branch	transitions	links
bigquery -> bigquery	34275/34695 passed (98.8%)	33123/33540 passed (98.8%)	2 pass -> fail	full result / delta
databricks -> databricks	10002/11820 passed (84.6%)	9999/11820 passed (84.6%)	3 pass -> fail	full result / delta

Overall

main: 192414 total, 153544 passed (pass rate: 79.8%)

sqlglot:optimizer/bug-query-star-collision: 180220 total, 142392 passed (pass rate: 79.0%)

Transitions:
5 pass -> fail

Dialect pair changes: 0 previous results not found, 3 current results not found

❌ 5 regressions (view logs)

fivetran-kwoodbeck · 2026-07-14T21:04:18Z

@fivetran-kwoodbeck I don't think we should bother dealing with this. These queries seem pathological and the amount of complexity and overhead I'm seeing in this PR makes me feel like this is definitely not worth it.

Postgres' behavior for example demonstrates, imo, why something like this in any normal system / pipeline would be pretty much unusable:
georgesittas=# SELECT * FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOIN t2 AS b) AS w;
 c | c | c
---+---+---
 1 | 1 | 2
(1 row)

georgesittas=# SELECT c FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOIN t2 AS b) AS w;
ERROR:  column reference "c" is ambiguous
LINE 1: SELECT c FROM t1 CROSS JOIN (SELECT * FROM t1 AS a CROSS JOI...
               ^

@georgesittas I hear you on the complexity, but this bug was detected by analyzing actual queries from Fivetran's production BigQuery history. It's a legit bug that will trigger in production. Just think of how often columns named id or name are used.

You're right about the explicit column case (SELECT c), engines should error out there. The problem is with the SELECT * path. Your first example actually runs perfectly across engines today, but the optimizer rewrites the query into either a query that's semantically different and might / not run:

DuckDB: Silently alters the output (returning 1, 1, 1 instead of 1, 1, 2).
BigQuery / Snowflake: Throws an ambiguous-column error, breaking a previously working query.

If the canonical output is hashed, it can also cause different queries to collide.

One idea is we could look for colliding output names in a single pass after the stars are expanded. I think that would simplify things.

georgesittas · 2026-07-20T13:36:12Z

+                    source.expression.set("expressions", [exp.Star()])
+                    del new_selections[star_start:]
+                    new_selections.append(expression)
+                    break


Why are we overwriting the whole expressions here with a single star? What happens when a projection list contains both stars and actual projections?

georgesittas · 2026-07-20T13:38:16Z

+            if isinstance(source, Scope) and isinstance(source.expression, exp.Select):
+                # This source is being re-exposed via a star, check for duplicate output
+                # and restore the star if found.
+                if _has_duplicate_output_names(source.expression):


This can also clobber duplicate aliases authored by the user explicitly. For example:

-- input, produces (k, k) SELECT * FROM (SELECT a AS k, a AS k FROM x) AS s -- output, produces (a, b) WITH "s" AS (SELECT * FROM "x" AS "x") SELECT * FROM "s"

georgesittas · 2026-07-20T13:39:27Z

+# title: outer star over derived table with duplicate output names is left unexpanded
+# execute: false
+SELECT * FROM (SELECT * FROM x CROSS JOIN y) AS s;
+SELECT * FROM (SELECT * FROM x AS x CROSS JOIN y AS y) AS s;


The testing coverage here is insufficient.

georgesittas · 2026-07-20T13:57:35Z

Isn't the solution as simple as extending this with or len(columns) != len(set(columns))?

That could work. I was aiming to not expand when there's duplicates. So, for example:

SELECT * FROM (SELECT * FROM x CROSS JOIN y) AS s

would now turn into this:

SELECT * FROM ( SELECT x.id AS id, x.name AS name, y.id AS id, y.name AS name FROM x CROSS JOIN y ) AS s;

It has (incorrectly named) duplicates, but it'll run and be idempotent which are the main points that matter.

I see, makes sense. My approach doesn't expand when there are duplicates within one source, but doesn't handle cross-source projection duplication.

I'm not sure what the best approach is, but I would check if it makes raising a qualify error if any duplicates are detected at any scope level, to avoid ambiguity from creeping up on us.

Up to you, I'm also happy to investigate if you have other things on your plate.

at any scope level

May be fine to not do this at the top-level scope. I think most engines will run such queries. The issue appears when you try to project from or expand a scope that contains duplicate projection names.

fivetran-kwoodbeck force-pushed the optimizer/bug-query-star-collision branch from 16ec962 to 7c53f32 Compare July 14, 2026 17:36

georgesittas reviewed Jul 14, 2026

View reviewed changes

fivetran-kwoodbeck force-pushed the optimizer/bug-query-star-collision branch from 7c53f32 to 842ccde Compare July 15, 2026 13:39

georgesittas reviewed Jul 16, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

fivetran-kwoodbeck force-pushed the optimizer/bug-query-star-collision branch from 842ccde to 79e4f04 Compare July 16, 2026 16:18

georgesittas reviewed Jul 17, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

fivetran-kwoodbeck force-pushed the optimizer/bug-query-star-collision branch 2 times, most recently from 3d198c0 to 2eccfcc Compare July 17, 2026 22:36

georgesittas self-assigned this Jul 20, 2026

georgesittas requested changes Jul 20, 2026

View reviewed changes

fivetran-kwoodbeck added 7 commits July 20, 2026 17:15

deal with uniqueness issue with colliding star expanded aliases

fd5d2a8

switched to single gated post pass

fd278c4

optimized star expansion bugfix

678fcbd

comment

5045139

switch over to non-expansion of duplicates

0e882a3

simplified merger test

efe1793

expanded tests

75954c5

fivetran-kwoodbeck force-pushed the optimizer/bug-query-star-collision branch from 2eccfcc to 75954c5 Compare July 20, 2026 21:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(optimizer): dedupe colliding star expanded aliases#7872

fix(optimizer): dedupe colliding star expanded aliases#7872
fivetran-kwoodbeck wants to merge 7 commits into
mainfrom
optimizer/bug-query-star-collision

fivetran-kwoodbeck commented Jul 14, 2026 •

edited

Loading

Uh oh!

georgesittas left a comment •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

fivetran-kwoodbeck commented Jul 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

georgesittas Jul 20, 2026

Uh oh!

georgesittas Jul 20, 2026 •

edited

Loading

Uh oh!

georgesittas Jul 20, 2026

Uh oh!

georgesittas Jul 20, 2026

Uh oh!

fivetran-kwoodbeck Jul 20, 2026 •

edited

Loading

Uh oh!

georgesittas Jul 20, 2026

Uh oh!

georgesittas Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fivetran-kwoodbeck commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

georgesittas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SQLGlot Integration Test Results

❌ 5 regressions — see details below

By Dialect

Overall

Uh oh!

fivetran-kwoodbeck commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

georgesittas Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgesittas Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

fivetran-kwoodbeck Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgesittas Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fivetran-kwoodbeck commented Jul 14, 2026 •

edited

Loading

georgesittas left a comment •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

fivetran-kwoodbeck commented Jul 14, 2026 •

edited

Loading

georgesittas Jul 20, 2026 •

edited

Loading

fivetran-kwoodbeck Jul 20, 2026 •

edited

Loading