Skip to content

Resolve cast-style CONVERT columns against the visible schema - #575

Merged
perNyfelt merged 3 commits into
mainfrom
fix/324-convert-type-named-column
Nov 18, 2026
Merged

perNyfelt merged 3 commits into
mainfrom
fix/324-convert-type-named-column

Conversation

@perNyfelt

Copy link
Copy Markdown
Member

Fixes #324.

Problem

For SQL Server-style CONVERT(<type>, <column>), JSqlParser puts the column in getExpression() and the type in getColDataType(). SqlParser.analyzeConvertFunction tried to undo value-first argument order by assuming that an inner expression spelling a type name is the target type. It never checked whether that name is actually a column in the table schema, so a real column named like a type (int8, int4, float8, bool, text, ...) made column extraction look for a column named after the target type instead:

Parquet table conv(id INT, int8 BIGINT, note STRING)
SELECT CONVERT(VARCHAR, int8) AS v FROM conv
  -> SQLSyntaxErrorException 42000: Missing columns for table 'conv': VARCHAR

ValueExpressionEvaluator already had the schema-column guard (looksLikeSchemaColumn via CastExpressions.asCast); column extraction did not.

Fix

  • Add SqlParser.analyzeConvertFunction(convert, schemaColumns): it resolves the CONVERT the same way CastExpressions.asCast does, so column extraction agrees with evaluation. When both arguments spell type names, a Column operand naming a visible source column keeps its value role. When the arguments are ambiguous (both name visible columns), the previous heuristic is retained so the evaluator's "Ambiguous CONVERT arguments" error still surfaces.
  • Thread a schema-column predicate (MaterializedNames.visibleColumns / lookupKeys) through ColumnsUsed.inWhere/inScope/referencesInScope, SqlParser.collectQualifiedColumns, and their call sites in JParqPreparedStatement (select, join output, join conditions, join tables, ambiguous-column checks, set-operation ORDER BY) and JParqResultSet (residual/HAVING/projection/window collection).
  • Existing overloads without schema knowledge keep the previous behavior exactly.

Tests

New ConvertTypeNameColumnTest (jparq package) with Parquet fixtures containing physical columns named int8, int4, float8, bool, text plus an ambiguous table with both varchar and int8:

  • both argument orders for every type-named column, compared against the equivalent CAST, across all four predicate/projection pushdown combinations;
  • the exact issue 324 repro queries (CONVERT(VARCHAR, int8), CONVERT(DOUBLE, int8), CONVERT(VARCHAR, note));
  • CONVERT in WHERE (residual), GROUP BY, ORDER BY, and JOIN ON clauses;
  • ambiguous CONVERT(VARCHAR, int8) against the table that has both columns still fails preparation with 42000 "Ambiguous CONVERT arguments" instead of a phantom missing column.

Verified the new tests fail on the pre-fix commit with the issue's exact error (8 of 9 cases) and pass with the fix. Full mvn verify (unit + IT + checkstyle + PMD + spotless + JaCoCo) passes on Java 21.

Also ignores .DS_Store in .gitignore (separate commit).

SqlParser.analyzeConvertFunction assumed that an inner expression spelling a
type name was the target type, so CONVERT(VARCHAR, int8) extracted a phantom
VARCHAR column whenever int8 was a real column named after a type. Mirror
CastExpressions.asCast with a schema-column predicate so extraction agrees
with evaluation, and thread the predicate through ColumnsUsed,
collectQualifiedColumns, and their validation call sites. When both
arguments name visible columns the legacy heuristic is retained, keeping the
ambiguous-CONVERT error from the evaluator.
@perNyfelt perNyfelt added the bug Something isn't working label Nov 18, 2026

@perNyfelt perNyfelt left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Existing feedback: none — no prior reviews, review threads, or PR comments (including from Copilot/bots), so nothing is outstanding.

Verification (local, Java 21.0.12):

  • Full mvn verify: BUILD SUCCESS — all unit + IT tests pass, 0 checkstyle violations, PMD and Spotless clean, JaCoCo check met.
  • New ConvertTypeNameColumnTest: 9/9 pass; CI verify is green as well.

Logic review:

  • SqlParser.analyzeConvertFunction(convert, schemaColumns) mirrors CastExpressions.asCast correctly: the non-swap branch sets the left expression to the same object as getExpression(), so the identity check cast.getLeftExpression() != inner reliably distinguishes cast-order from value-first order, and the swap branch's new Column(candidate) renders back to exactly the previous typeCandidate. I checked the no-schema overload case-by-case (cast-order, value-first, both-args-type-named, function operands, ambiguous fallback) and it is behaviorally identical to the old heuristic, which is also confirmed by the existing CONVERT tests.
  • MaterializedNames.visibleColumns/lookupKeys are null-safe (lookupIndex(null) returns an empty index) and apply Identifier.lookupKey on both stored keys and probes, matching the evaluator's looksLikeSchemaColumn.
  • The predicate threading is consistent across JParqResultSet (residual/HAVING/projection/window), JParqPreparedStatement (required columns, ambiguity checks, join validation, set-operation ORDER BY), ColumnsUsed, and collectQualifiedColumns. validateTableFreeSelect correctly keeps name -> false (no FROM, no schema), and the remaining old-overload call in normalizeScopeQualifiers (SqlParser.java:3626) is fine: synthetic columns carry no qualifier and table schemas are unknown at parse time.

Findings:

  1. Minor (new checkstyle warning): JParqPreparedStatement.java:3254 — VariableDeclarationUsageDistance: the schemaColumns declaration in validateJoinOutputColumns is 4 lines from its first use (allowed 3). Severity is warning so the build passes, but this PR introduces a new warning (the other one, JParqResultSet.java:1934 OverloadMethodsDeclarationOrder, is pre-existing — those getObject overloads are untouched by this PR). Suggestion: move the declaration to just before the for (Expression expression : expressions) loop.

  2. Observation (pre-existing, no action required): the preparation walk at SqlParser.java:863 still resolves CONVERT via the no-schema asCast, so DecimalCasts.validateBounds/binaryLength are skipped for CONVERTs that are resolvable with schema knowledge (e.g. CONVERT(int8, VARCHAR) where int8 is a real column). Value-level bounds are still enforced at evaluation, so this is validation timing only; threading the predicate there could be a follow-up.

Tests: good coverage — both argument orders for every type-named column, all four pushdown combinations, WHERE/GROUP BY/ORDER BY/JOIN ON, and the 42000 ambiguous-arguments case.

Overall: the fix is correct, minimal, and well tested. The only actionable item is finding 1.

@perNyfelt
perNyfelt merged commit 4b81a00 into main Nov 18, 2026
2 checks passed
@perNyfelt
perNyfelt deleted the fix/324-convert-type-named-column branch November 18, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CONVERT(type, column) fails with 'Missing columns' when the column name is also a type name

1 participant