Repository navigation
Fix tree-model NOT predicate pushdown #18982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8643565
42867e3
34c39ad
1b9ddcf
dd072c7
3c33cc4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1547,7 +1547,7 @@ private void analyzeDeviceToWhere( | |
| while (deviceIterator.hasNext()) { | ||
| PartialPath devicePath = deviceIterator.next(); | ||
| Expression whereExpression = | ||
| analyzeWhereSplitByDevice(queryStatement, devicePath, schemaTree, queryContext); | ||
| analyzeWhereSplitByDevice(analysis, queryStatement, devicePath, schemaTree, queryContext); | ||
| if (whereExpression.equals(ConstantOperand.FALSE)) { | ||
| deviceIterator.remove(); | ||
| } else if (whereExpression.equals(ConstantOperand.TRUE)) { | ||
|
|
@@ -1580,7 +1580,7 @@ private void analyzeWhere( | |
| schemaTree, | ||
| true, | ||
| queryContext); | ||
| Expression whereExpression = convertConJunctionsToWhereExpression(conJunctions); | ||
| Expression whereExpression = convertConJunctionsToWhereExpression(analysis, conJunctions); | ||
| if (whereExpression.equals(ConstantOperand.TRUE)) { | ||
| analysis.setWhereExpression(null); | ||
| analysis.setHasValueFilter(false); | ||
|
|
@@ -1595,6 +1595,7 @@ private void analyzeWhere( | |
| } | ||
|
|
||
| private Expression analyzeWhereSplitByDevice( | ||
| final Analysis analysis, | ||
| final QueryStatement queryStatement, | ||
| final PartialPath devicePath, | ||
| final ISchemaTree schemaTree, | ||
|
|
@@ -1606,16 +1607,25 @@ private Expression analyzeWhereSplitByDevice( | |
| schemaTree, | ||
| true, | ||
| queryContext); | ||
| return convertConJunctionsToWhereExpression(conJunctions); | ||
| return convertConJunctionsToWhereExpression(analysis, conJunctions); | ||
| } | ||
|
|
||
| private Expression convertConJunctionsToWhereExpression(List<Expression> conJunctions) { | ||
| private Expression convertConJunctionsToWhereExpression( | ||
| Analysis analysis, List<Expression> conJunctions) { | ||
| Expression predicate = | ||
| PredicateUtils.combineConjuncts( | ||
| conJunctions.stream().distinct().collect(Collectors.toList())); | ||
| predicate = PredicateUtils.simplifyPredicate(predicate); | ||
| predicate = normalizeExpression(predicate); | ||
| return predicate; | ||
| analyzeExpressionType(analysis, predicate); | ||
| Expression predicateWithoutNot = | ||
| PredicateUtils.predicateRemoveNot(predicate, analysis::getType); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now that When the plan has a Repro (same setup as my first review): SELECT s1, s2 FROM root.rv.d1 WHERE NOT (time > 2 OR s1 > 5)
-- expected: (1, 1, 10) this PR: 305 ... executeStatement failed. null
SELECT s1, s2 FROM root.rv.d1 WHERE NOT (time > 2 OR s1 > 5) ALIGN BY DEVICE
-- expected: (1, root.rv.d1, 1, 10) this PR: 305 ... null
SELECT s1, s2 FROM root.rv.d1 WHERE NOT (time > 2 OR s1 > 5) AND s2 > 5
-- expected: (1, 1, 10) this PR: 305 ... null
SELECT s1, s2 FROM root.rv.d1 WHERE NOT ((time > 2 AND time < 4) OR s1 > 5)
-- expected: (1, 1, 10) this PR: 305 ... nullThese queries only work today in two cases:
On master these queries fail with 301 The underlying bug already exists on master. A time-only conjunct that can't be extracted as a global time filter crashes the same way there, e.g. Please also add an IT that reads at least two non-aligned series, e.g. |
||
| if (predicateWithoutNot != predicate) { | ||
| predicateWithoutNot = PredicateUtils.simplifyPredicate(predicateWithoutNot); | ||
| predicateWithoutNot = normalizeExpression(predicateWithoutNot); | ||
| analyzeExpressionType(analysis, predicateWithoutNot); | ||
| } | ||
| return predicateWithoutNot; | ||
| } | ||
|
|
||
| private void analyzeDeviceViewOutput(Analysis analysis, QueryStatement queryStatement) { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both
HAVING NOT (count(int32_col) > 1)cases (with and without ALIGN BY DEVICE) already pass on master, so they don't cover the HAVING change. The HAVING bug this PR actually fixes is a compositeNOTon the non-ALIGN BY DEVICE and GROUP BY LEVEL paths (see the comment onanalyzeHavingBase).It would be good to add:
HAVING NOT (count(...) > 1 AND sum(...) > 50), with and withoutGROUP BY LEVELNOT:WHERE NOT bool_series,WHERE NOT bool_series OR s1 > 5,HAVING NOT last_value(bool_series)WHERE NOT (NOT (NOT (A AND B)))NOTAny of the last three would have caught the issues raised in the other comments.
PredicateUtilsTest#testLogicNotCannotPushDownToSourcecovers only the checker, not the analyzer path.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the requested coverage: composite HAVING with and without GROUP BY LEVEL, non-invertible boolean series/functions/CASE, triple negation, NaN, and aligned-template predicates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The missing
HAVING NOT last_value(bool_series)coverage is added now — that was the one item from your list I had missed. Added in 1b9ddcf:testHavingNotWithBooleanAggregation:HAVING NOT last_value(b)withGROUP BY ([1,5),1ms), plus the same query withALIGN BY DEVICE.testTemplatedHavingNotWithBooleanAggregation: the aligned-template path,SELECT count(s1) FROM root.test_not_template.* GROUP BY ([1,3),1ms) HAVING NOT last_value(b) ALIGN BY DEVICE.The other cases you listed were already in the PR: composite HAVING with and without
GROUP BY LEVEL,WHERE NOT bool_series,WHERE NOT bool_series OR s1 > 5,NOT (CASE WHEN ...) OR ..., triple negation and the NaN cases.Verified locally on the latest head:
PredicateUtilsTest13/13 andIoTDBPredicateConversionTreeIT11/11 (including the two new tests).