PS-9768 [8.4] duplicate key in table while using tmp table for group by a json data - #6118
Draft
catalinbp wants to merge 4 commits into
Draft
PS-9768 [8.4] duplicate key in table while using tmp table for group by a json data#6118catalinbp wants to merge 4 commits into
catalinbp wants to merge 4 commits into
Conversation
https://perconadev.atlassian.net/browse/PS-9768 A grouped query over a JSON column failed with ERROR 23000: Can't write; duplicate key in table '/.../#sql...' when the table had an indexed stored generated column unquoting the very same expression: a varchar(128) GENERATED ALWAYS AS (json_unquote(json_extract(data2,'$.a'))) STORED, KEY (a,i) SELECT json_extract(data2,'$.a'), CAST(count(*) AS JSON) FROM t WHERE i = 1 GROUP BY json_extract(data2,'$.a'); At optimize time substitute_gc() replaces expressions in the WHERE condition and in the ORDER/GROUP BY lists with equivalent indexed generated columns. The matching is done by get_gc_for_expr(), which deliberately ignores a JSON_UNQUOTE() wrapper on the GC expression when the analysed expression does not have one. That provision exists so that a predicate such as WHERE json_extract(data2,'$.a') = 'a' can be evaluated through the index over the unquoting GC, and its comment scopes it explicitly to the WHERE condition. JSON_UNQUOTE() however changes the value and not only its representation for comparison purposes: json_extract(data2,'$.a') evaluates to '"a"' while the GC holds 'a'. GROUP BY consumes the value of the expression itself, so the grouping temporary table was filled with the unquoted GC value while the lookup preceding the insert used the original quoted expression value. The mismatch made the executor insert the same key twice instead of finding it and aggregating COUNT, hence the duplicate key error. Beyond the error this is a wrong-results problem: the distinct JSON values 1 and "1" both unquote to '1' and would be merged into a single group, and ORDER BY is affected as well since JSON comparison sorts numbers before strings whereas the unquoted string ordering does not. Only substitutions that preserve the value of the expression are correct for the ORDER/GROUP BY lists. - Add a `require_same_value` parameter to get_gc_for_expr(). When set, the JSON_UNQUOTE() wrapper of the GC expression is no longer skipped while matching, so a GC that unquotes the expression is not equivalent to it. - Pass require_same_value=true from the ORDER/GROUP BY substitution loop in substitute_gc() and document the restriction (sql/sql_optimizer.cc). The WHERE condition call sites (substitute_gc_expression() and gc_subst_overlaps_contains()) keep the lenient matching, so predicate pushdown to an index over an unquoting GC is unaffected. As the substitution runs before the traditional/hypergraph planner split, both optimizers are fixed, as are the single-table UPDATE/DELETE ORDER BY call sites in sql_update.cc and sql_delete.cc. Regression coverage: - Add percona.ps9768 covering the reported query, the equivalent ORDER BY and DELETE ... ORDER BY cases, that an exact GROUP BY match against the GC expression is still substituted, and that WHERE pushdown still is. Substitution is asserted through information_schema.OPTIMIZER_TRACE rather than EXPLAIN, to avoid row-estimate and hypergraph differences.
https://perconadev.atlassian.net/browse/PS-9768 The previous commit stopped substitute_gc() from replacing an ORDER/GROUP BY expression with a generated column that unquotes it. Two further substitutions performed by the same loop are not value preserving either and silently return wrong results. 1) A multi-valued index's field holds the set of values the expression evaluates to and not the value itself, yet it was substituted into the GROUP BY list because the CAST(.. AS .. ARRAY) wrapper of its expression is skipped while matching: CREATE TABLE t (j json, INDEX mvi ((CAST(j->'$.arr' AS UNSIGNED ARRAY)))); INSERT INTO t VALUES ('{"arr":[1,2,3]}'), ('{"arr":[4,5,6]}'); SELECT j->'$.arr', count(*) FROM t GROUP BY j->'$.arr'; -> [1, 2, 3] 2 Two distinct arrays have to be two groups, as returned once the index is made unusable with IGNORE KEY FOR GROUP BY. 2) Grouping and sorting are performed according to the collation, so a GC whose collation differs from the one of the expression does not group or order the rows the same way. JSON_UNQUOTE() returns utf8mb4_bin, while a GC or functional index over it takes the table's collation, which is case insensitive by default: CREATE TABLE t (data2 json, INDEX idx ((CAST(data2->>'$.a' AS CHAR(30))))); INSERT INTO t VALUES (JSON_OBJECT('a','a')), (JSON_OBJECT('a','A')); SELECT data2->>'$.a', count(*) FROM t GROUP BY data2->>'$.a'; -> a 2 'a' and 'A' are two groups for utf8mb4_bin, and the equivalent ORDER BY is sorted case insensitively as well. - Remove multi-valued index fields from the list of candidates used for the ORDER/GROUP BY lists. - Require the collation of a string GC to match the collation of the expression it would replace. The WHERE condition only needs this for functional indexes (see bug#27337092 and substitute_gc_expression()), because there the GC is merely used to look up rows, whereas the value of an ORDER/GROUP BY expression is consumed by the executor. Both restrictions apply to the ORDER/GROUP BY substitution only, so predicate pushdown is unchanged: the multi-valued index is still used for MEMBER OF, JSON_CONTAINS and JSON_OVERLAPS, and the functional index is still used for an equality on the CAST() expression. Substitutions that do preserve the value keep working, e.g. GROUP BY on the exact functional index expression, on a GC declared with the matching collation, or on a numeric GC.
https://perconadev.atlassian.net/browse/PS-9768 require_same_value already stopped get_gc_for_expr() from skipping a JSON_UNQUOTE() wrapper when matching a GC for ORDER/GROUP BY. CAST() was still skipped, so a functional index could replace a non-equivalent expression and silently return wrong results: CREATE TABLE t (j json, INDEX idx ((CAST(j->>'$.n' AS UNSIGNED)))); INSERT INTO t VALUES (JSON_OBJECT('n','1')), (JSON_OBJECT('n','01')); SELECT j->>'$.n', count(*) FROM t GROUP BY j->>'$.n'; -> 1 2 '1' and '01' are two groups; CAST turns both into the integer 1. The same hole exists for a length-changing CAST with a matching collation, which merges 'aa' and 'ab' under CAST(.. AS CHAR(1)). - When require_same_value is set, also refuse to strip TYPECAST_FUNC. Exact GROUP BY/ORDER BY on the CAST() expression still matches, because the wrapper is only skipped when the analysed expression lacks it. - COLLATE continues to be stripped; collation mismatches remain handled in substitute_gc(). WHERE pushdown is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.