Repository navigation
Resolve cast-style CONVERT columns against the visible schema - #575
Conversation
SqlParser.analyzeConvertFunction assumed that an inner expression spelling a type name was the target type, so CONVERT(VARCHAR, int8) extracted a phantom VARCHAR column whenever int8 was a real column named after a type. Mirror CastExpressions.asCast with a schema-column predicate so extraction agrees with evaluation, and thread the predicate through ColumnsUsed, collectQualifiedColumns, and their validation call sites. When both arguments name visible columns the legacy heuristic is retained, keeping the ambiguous-CONVERT error from the evaluator.
perNyfelt
left a comment
There was a problem hiding this comment.
Review
Existing feedback: none — no prior reviews, review threads, or PR comments (including from Copilot/bots), so nothing is outstanding.
Verification (local, Java 21.0.12):
- Full
mvn verify: BUILD SUCCESS — all unit + IT tests pass, 0 checkstyle violations, PMD and Spotless clean, JaCoCo check met. - New
ConvertTypeNameColumnTest: 9/9 pass; CIverifyis green as well.
Logic review:
SqlParser.analyzeConvertFunction(convert, schemaColumns)mirrorsCastExpressions.asCastcorrectly: the non-swap branch sets the left expression to the same object asgetExpression(), so the identity checkcast.getLeftExpression() != innerreliably distinguishes cast-order from value-first order, and the swap branch'snew Column(candidate)renders back to exactly the previoustypeCandidate. I checked the no-schema overload case-by-case (cast-order, value-first, both-args-type-named, function operands, ambiguous fallback) and it is behaviorally identical to the old heuristic, which is also confirmed by the existing CONVERT tests.MaterializedNames.visibleColumns/lookupKeysare null-safe (lookupIndex(null)returns an empty index) and applyIdentifier.lookupKeyon both stored keys and probes, matching the evaluator'slooksLikeSchemaColumn.- The predicate threading is consistent across JParqResultSet (residual/HAVING/projection/window), JParqPreparedStatement (required columns, ambiguity checks, join validation, set-operation ORDER BY), ColumnsUsed, and collectQualifiedColumns.
validateTableFreeSelectcorrectly keepsname -> false(no FROM, no schema), and the remaining old-overload call innormalizeScopeQualifiers(SqlParser.java:3626) is fine: synthetic columns carry no qualifier and table schemas are unknown at parse time.
Findings:
-
Minor (new checkstyle warning): JParqPreparedStatement.java:3254 —
VariableDeclarationUsageDistance: theschemaColumnsdeclaration invalidateJoinOutputColumnsis 4 lines from its first use (allowed 3). Severity is warning so the build passes, but this PR introduces a new warning (the other one, JParqResultSet.java:1934OverloadMethodsDeclarationOrder, is pre-existing — thosegetObjectoverloads are untouched by this PR). Suggestion: move the declaration to just before thefor (Expression expression : expressions)loop. -
Observation (pre-existing, no action required): the preparation walk at SqlParser.java:863 still resolves CONVERT via the no-schema
asCast, soDecimalCasts.validateBounds/binaryLengthare skipped for CONVERTs that are resolvable with schema knowledge (e.g.CONVERT(int8, VARCHAR)whereint8is a real column). Value-level bounds are still enforced at evaluation, so this is validation timing only; threading the predicate there could be a follow-up.
Tests: good coverage — both argument orders for every type-named column, all four pushdown combinations, WHERE/GROUP BY/ORDER BY/JOIN ON, and the 42000 ambiguous-arguments case.
Overall: the fix is correct, minimal, and well tested. The only actionable item is finding 1.
Fixes #324.
Problem
For SQL Server-style
CONVERT(<type>, <column>), JSqlParser puts the column ingetExpression()and the type ingetColDataType().SqlParser.analyzeConvertFunctiontried to undo value-first argument order by assuming that an inner expression spelling a type name is the target type. It never checked whether that name is actually a column in the table schema, so a real column named like a type (int8,int4,float8,bool,text, ...) made column extraction look for a column named after the target type instead:ValueExpressionEvaluatoralready had the schema-column guard (looksLikeSchemaColumnviaCastExpressions.asCast); column extraction did not.Fix
SqlParser.analyzeConvertFunction(convert, schemaColumns): it resolves the CONVERT the same wayCastExpressions.asCastdoes, so column extraction agrees with evaluation. When both arguments spell type names, aColumnoperand naming a visible source column keeps its value role. When the arguments are ambiguous (both name visible columns), the previous heuristic is retained so the evaluator's "Ambiguous CONVERT arguments" error still surfaces.MaterializedNames.visibleColumns/lookupKeys) throughColumnsUsed.inWhere/inScope/referencesInScope,SqlParser.collectQualifiedColumns, and their call sites inJParqPreparedStatement(select, join output, join conditions, join tables, ambiguous-column checks, set-operation ORDER BY) andJParqResultSet(residual/HAVING/projection/window collection).Tests
New
ConvertTypeNameColumnTest(jparq package) with Parquet fixtures containing physical columns namedint8,int4,float8,bool,textplus an ambiguous table with bothvarcharandint8:CONVERT(VARCHAR, int8),CONVERT(DOUBLE, int8),CONVERT(VARCHAR, note));CONVERT(VARCHAR, int8)against the table that has both columns still fails preparation with 42000 "Ambiguous CONVERT arguments" instead of a phantom missing column.Verified the new tests fail on the pre-fix commit with the issue's exact error (8 of 9 cases) and pass with the fix. Full
mvn verify(unit + IT + checkstyle + PMD + spotless + JaCoCo) passes on Java 21.Also ignores
.DS_Storein .gitignore (separate commit).