diff --git a/CHANGELOG.md b/CHANGELOG.md index afc861fe..03fa3396 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -227,6 +227,24 @@ true until the next version shipped. ### Changed +- `GROUP BY` is vectorized by default (#1292). + `pgcolumnar.enable_group_vectorization` is now `on`. The planner takes the + grouped path only when an upper bound on the group count is within + `pgcolumnar.groupagg_max_groups`: the table's live row count, or, for + time-bucket keys, the buckets #369's and #1304's bounds allow. Above that the + ordinary plan runs. Before, the path was off because over the cap it errors at + execution, and on by default that error would have reached any high-cardinality + `GROUP BY`. The first version gated on the estimate, and @OffgridwithJD's review + showed a skewed key fooling ANALYZE by 68x with fresh statistics (29,356 + estimated for 2,000,001) and erroring by default; a bound cannot be fooled so. + The runtime cap stays as the backstop for a cap lowered, or a table grown, + after planning; `native_groupagg` shows it with a cached plan. Three suites assumed the old default: `native_groupagg` forced the + node under a cap of 100, which the gate now declines; `groupagg_table_sizing` + raises the cap for its 200,000-group fixture; and `vector_agg_tlist_shape` + pins the grouped path off, because it tests the ungrouped fold. The docs said + the path took only plain columns and `count`, `sum` and `avg`; it also takes + `date_bin`, `date_trunc`, `min` and `max`. + - The planner estimates the number of groups for a time-bucket key from the table's own data (#1292). `GROUP BY date_trunc(unit, ts)` and, newly, `GROUP BY date_bin(stride, ts, origin)` were estimated near the row count diff --git a/docs/configuration.md b/docs/configuration.md index 2b12a5f7..96796e8f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -59,8 +59,8 @@ disk. It never changes the values that a table returns. | `pgcolumnar.enable_custom_scan` | boolean | `on` | Use the columnar custom scan path for columnar tables. | | `pgcolumnar.enable_qual_pushdown` | boolean | `on` | Push scan qualifiers down so per-chunk min and max values can skip chunk groups. | | `pgcolumnar.enable_vectorization` | boolean | `on` | Use the vectorized aggregate path for supported ungrouped aggregates. | -| `pgcolumnar.enable_group_vectorization` | boolean | `off` | Use the vectorized aggregate path for `GROUP BY` queries on a columnar table. Off by default; see [why grouped vectorization is off by default](#why-grouped-vectorization-is-off-by-default). | -| `pgcolumnar.groupagg_max_groups` | integer | `1000000` | Cap on the group count the grouped vectorized aggregate builds. Over the cap the query errors. Range 1 to INT_MAX. | +| `pgcolumnar.enable_group_vectorization` | boolean | `on` | Use the vectorized aggregate path for `GROUP BY` queries on a columnar table. The planner takes it only when the group count cannot exceed `pgcolumnar.groupagg_max_groups`; see [how the grouped path stays safe on by default](#how-the-grouped-path-stays-safe-on-by-default). | +| `pgcolumnar.groupagg_max_groups` | integer | `1000000` | Cap on the group count the grouped vectorized aggregate builds. Over the cap the query errors. The planner takes the grouped path only when the group count cannot exceed this. Range 1 to INT_MAX. | | `pgcolumnar.enable_bloom_filter` | boolean | `on` | Skip chunk groups on equality filters using per-chunk bloom filters. | | `pgcolumnar.enable_join_runtime_filter` | boolean | `on` | Wrap a serial inner Hash Join so the build keys can skip fact-table groups and reject non-matching rows. Direct columnar scan only. Group skip needs the fact table clustered on the join key. On by default. | | `pgcolumnar.enable_read_stream` | boolean | `on` | Prefetch block reads with the read stream API. Effective on PostgreSQL 17 and later. | @@ -192,7 +192,7 @@ limit or a level. clear a declared retention, set `ttl_interval` to a value long enough that no row reaches it, or recreate the table. -### Why grouped vectorization is off by default +### How the grouped path stays safe on by default `pgcolumnar.enable_group_vectorization` is a real speed-up. Take 4,000,000 rows grouped into 200,000 keys. `SELECT k, sum(m) FROM t GROUP BY k` runs in 403.7 ms @@ -204,8 +204,8 @@ reset, and the ordinary `Agg` resets far more contexts than the vectorized fold does. Your own factor depends on the build, the group count, the key width and the aggregate. -It stays off because of how it fails, not because of how it performs. The -grouped path builds a hash table that does not spill, so +Until 1.0-alpha5 it was off by default, because of how it fails, not how it +performs. The grouped path builds a hash table that does not spill, so `pgcolumnar.groupagg_max_groups` bounds it at 1,000,000 groups. The cap is checked during execution, against the real group count. By then the plan is fixed, and the path cannot hand the query back to an ordinary `Agg`. Over the @@ -216,14 +216,22 @@ ERROR: 54000: grouped vectorized aggregate exceeded pgcolumnar.groupagg_max_gro HINT: Raise pgcolumnar.groupagg_max_groups, or set pgcolumnar.enable_group_vectorization = off. ``` -Measured on a table with 1,500,000 distinct keys: the same query succeeds with -the setting off and raises that error with it on. +On by default, that error would reach tables nobody chose the path for. So the +planner now takes the grouped path only when the group count cannot exceed the +cap. It uses an upper bound, not an estimate: -A default carries that behaviour to tables the user did not choose it for. The -failure appears when a table grows past the cap. A query that ran yesterday then -fails today, with no change to the query and no change to the setting. Turn the -setting on for a workload whose group count you know. Raise the cap for one that -approaches it. +- the table's live row count, because no grouping makes more groups than rows; +- for `date_bin` and `date_trunc` keys, the number of buckets the time column's + range allows, which is usually far smaller. + +A query over more rows than the cap, grouped by a plain key, runs the ordinary +plan and succeeds. An estimate could not be the gate. On a key where one value +fills half the table and the rest are unique, a fresh `ANALYZE` estimated 29,356 +groups for 2,000,001. + +The error remains possible only when something changes after planning: the cap +is lowered, or the table grows past it. A prepared statement can see this. To keep larger +tables on the fast path, raise `pgcolumnar.groupagg_max_groups`. ### Encode effort diff --git a/docs/features.md b/docs/features.md index e7f7c7b3..5e869fef 100644 --- a/docs/features.md +++ b/docs/features.md @@ -76,11 +76,16 @@ stronger codec writes less. Neither path uses the per-tuple executor path. `count(*)` with no filter is one case of this: it is answered from each row group's stored row count and reads no column data. Set `pgcolumnar.enable_vectorization` to `off` to force an ordinary - aggregate over the scan instead. An opt-in path vectorizes `GROUP BY` too. Set - `pgcolumnar.enable_group_vectorization` to `on` to group and aggregate in one - pass over the reader. It is off by default. That pass applies where the - query groups by plain columns, aggregates with `count`, `sum` or `avg`, and - filters only with comparisons the scan keys express. There it reads each + aggregate over the scan instead. `GROUP BY` is vectorized too, by default. The + scan groups and aggregates in one pass over the reader. The planner takes this + path only when the groups cannot exceed `pgcolumnar.groupagg_max_groups`. The + table's row count bounds them, and so do the time buckets for a `date_bin` or + `date_trunc` key. It takes + plain columns and `date_bin` or `date_trunc` keys, and `count`, `sum`, `avg`, + `min` and `max`. Set `pgcolumnar.enable_group_vectorization` to `off` to turn + it off. Within it, the batch fold needs plain-column + keys and `count`, `sum` or `avg`. It also needs a filter of only the + comparisons the scan keys express. There it reads each column's values directly instead of one row at a time. `EXPLAIN` reports `Columnar Batch Fold` on the node. Other shapes read row at a time and return the same results. diff --git a/docs/how-to.md b/docs/how-to.md index bae9a8ba..6e73f50f 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -906,17 +906,19 @@ limitations guide. ## Vectorize a GROUP BY aggregate The vectorized aggregate path computes an ungrouped aggregate over decoded -vectors. An opt-in setting extends it to `GROUP BY`. +vectors. It also covers `GROUP BY`, on by default, when the planner estimates a +bounded number of groups. ```sql -SET pgcolumnar.enable_group_vectorization = on; EXPLAIN (ANALYZE) SELECT region, sum(amount) FROM events GROUP BY region; ``` -**Tuning.** Turn it on for a `GROUP BY` with a bounded number of distinct groups. -The plan then shows `Columnar Vectorized Group Keys`. A query that exceeds -`pgcolumnar.groupagg_max_groups` (default 1000000) raises an error, so leave the -setting off for high-cardinality grouping. +**Tuning.** The plan shows `Columnar Vectorized Group Keys` when the path is +taken. The planner takes it only when the group count cannot exceed +`pgcolumnar.groupagg_max_groups` (default 1000000). For a plain key the bound is +the table's row count, so a table of more rows than the cap gets the ordinary +plan. For time-bucket keys the bound is the number of buckets. Raise the cap to +keep larger tables on the fast path. ## Count rows quickly diff --git a/docs/limitations.md b/docs/limitations.md index 73c68ed1..c5fcbd4c 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -826,17 +826,19 @@ A dimension with duplicate keys stays on the core Agg plan. A LEFT join stays on the core Agg plan. Grouped aggregation over a join is not this path. -A separate opt-in path vectorizes `GROUP BY`. It is off by default. Set -`pgcolumnar.enable_group_vectorization` to `on` to enable it. It covers +A separate path vectorizes `GROUP BY`. It is on by default +(`pgcolumnar.enable_group_vectorization`). The planner takes it only when the +group count cannot exceed `pgcolumnar.groupagg_max_groups`. The table's row count +bounds it, and so do the time buckets for a time-bucket key. It covers `SELECT , agg(col) ... [WHERE ...] GROUP BY ` on one columnar relation. Each key must be non-volatile, hashable, and computable from that relation. A collatable key must use a deterministic collation. Each output must be a supported aggregate or a bare key reference. This path also accepts `sum` and `avg` on `bigint`, `numeric`, and floating-point columns, which the ungrouped path rejects. `pgcolumnar.groupagg_max_groups` caps the group count, default -1,000,000. The cap is checked at execution against the real count, so a query -that exceeds it errors rather than switching plans. Raise the cap or turn the -path off. +1,000,000. The cap is checked again at execution against the real count. A +plan made before the cap was lowered, or before the table grew past it, errors +there; it cannot switch plans. Raise the cap, or turn the path off. ## Join runtime filter diff --git a/src/columnar_tableam.c b/src/columnar_tableam.c index 3ccdc5ea..6f4f211d 100644 --- a/src/columnar_tableam.c +++ b/src/columnar_tableam.c @@ -4072,9 +4072,12 @@ _PG_init(void) DefineCustomBoolVariable("pgcolumnar.enable_group_vectorization", "Use the vectorized aggregate fast path for GROUP BY queries.", - NULL, + "Taken only when an upper bound on the group count (the " + "row count, or the time buckets) is within " + "pgcolumnar.groupagg_max_groups; otherwise the ordinary " + "plan runs.", &pgcolumnar_enable_group_vectorization, - false, + true, PGC_USERSET, 0, NULL, NULL, NULL); @@ -4104,10 +4107,13 @@ _PG_init(void) DefineCustomIntVariable("pgcolumnar.groupagg_max_groups", "Cap on the actual group count the grouped vectorized " "aggregate builds before it stops with an error.", - "Enforced at execution against the real number of groups, " - "not the planner's estimate: over the cap the query errors " - "rather than falling back, since the plan is fixed by then. " - "Raise it, or turn off pgcolumnar.enable_group_vectorization.", + "Enforced at execution against the real number of groups: " + "over the cap the query errors rather than falling back, " + "since the plan is fixed by then. The planner takes the " + "grouped path only when an upper bound on the group count " + "is within it, so the error means it fell, or the table " + "grew, after planning. Raise it, or turn off " + "pgcolumnar.enable_group_vectorization.", &pgcolumnar_groupagg_max_groups, 1000000, 1, INT_MAX, PGC_USERSET, diff --git a/src/columnar_vector.c b/src/columnar_vector.c index 81c82fb3..7444f2e8 100644 --- a/src/columnar_vector.c +++ b/src/columnar_vector.c @@ -92,13 +92,15 @@ bool pgcolumnar_enable_vectorization = true; /* - * GUC: extend the vectorized aggregate to GROUP BY (#289). Default off while the - * grouped path is built out incrementally; the ungrouped path is unaffected. - * groupagg_max_groups caps the ACTUAL group count at execution time; exceeding - * it raises an error (see pgcolumnar_groupagg_lookup) rather than routing to core - * HashAgg. It is not a plan-time gate. + * GUC: extend the vectorized aggregate to GROUP BY (#289). On by default since + * #1292 phase 2, because the planner now takes the path only when an UPPER + * BOUND on the group count is within groupagg_max_groups (see + * pgcolumnar_group_count_upper_bound). groupagg_max_groups still caps the + * ACTUAL group count at execution time; exceeding it raises an error (see + * pgcolumnar_groupagg_lookup) rather than routing to core HashAgg, which now + * needs the cap lowered, or the table grown past it, after planning. */ -bool pgcolumnar_enable_group_vectorization = false; +bool pgcolumnar_enable_group_vectorization = true; int pgcolumnar_groupagg_max_groups = 1000000; /* @@ -2367,13 +2369,12 @@ pgcolumnar_truncating_time_bound(PlannerInfo *root, RelOptInfo *input_rel, * A query with no such bound gets no bound from us. */ static double -pgcolumnar_group_estimate_bound(PlannerInfo *root, RelOptInfo *input_rel, - List *groupExprs) +pgcolumnar_group_estimate_bound(PlannerInfo *root, List *groupExprs, + double timeProduct) { ListCell *lc; - double bound = 1.0; - if (groupExprs == NIL) + if (groupExprs == NIL || timeProduct <= 0.0 || timeProduct >= (double) INT_MAX) return 0.0; foreach(lc, groupExprs) @@ -2381,7 +2382,6 @@ pgcolumnar_group_estimate_bound(PlannerInfo *root, RelOptInfo *input_rel, Node *expr = (Node *) lfirst(lc); VariableStatData vardata; bool informed; - double one; /* * Core's own question: does anything actually know about this expression? @@ -2392,16 +2392,64 @@ pgcolumnar_group_estimate_bound(PlannerInfo *root, RelOptInfo *input_rel, ReleaseVariableStats(vardata); if (informed) return 0.0; + } + return timeProduct; +} - one = pgcolumnar_truncating_time_bound(root, input_rel, expr); - if (one <= 0.0) - return 0.0; /* one unbounded key and the product is unbounded */ +/* + * pgcolumnar_time_bucket_products + * Each key's time-bucket bound (pgcolumnar_truncating_time_bound), read ONCE + * per plan, because each reads the zone maps (#1304: once per plan, not once + * per row group, and not twice). *product is their product, and *withNull + * the product with one more group per key for NULL; both are 0 when any key + * has no bound. + */ +static void +pgcolumnar_time_bucket_products(PlannerInfo *root, RelOptInfo *input_rel, + List *groupExprs, double *product, double *withNull) +{ + ListCell *lc; - bound *= one; - if (bound >= (double) INT_MAX) - return 0.0; + *product = 1.0; + *withNull = 1.0; + foreach(lc, groupExprs) + { + double one = pgcolumnar_truncating_time_bound(root, input_rel, + (Node *) lfirst(lc)); + + if (one <= 0.0) + { + *product = 0.0; + *withNull = 0.0; + return; + } + *product *= one; + *withNull *= one + 1.0; } - return bound; +} + +/* + * pgcolumnar_group_count_upper_bound + * A TRUE upper bound on the number of groups, for the gate (#1292 phase 2). + * + * The relation's live row count always bounds it: no grouping makes more + * groups than rows. input_rel->tuples is that count, read from the + * row-group catalog and the delete vectors (relation_estimate_size), not + * ANALYZE's sample, and before any WHERE, whose selectivity is an estimate. + * + * Where EVERY key is a truncating time function with a bound from its + * range (#369's quals, #1304's zone maps), the product of those bounds, + * with one more group per key for NULL, is tighter (timeWithNull, from + * pgcolumnar_time_bucket_products). Unlike pgcolumnar_group_estimate_bound, + * this applies even to a key that has statistics, because statistics are + * an estimate and the buckets a bound. + */ +static double +pgcolumnar_group_count_upper_bound(RelOptInfo *input_rel, double timeWithNull) +{ + double rows = Max(input_rel->tuples, 0.0); + + return (timeWithNull > 0.0 && timeWithNull < rows) ? timeWithNull : rows; } static void @@ -2417,6 +2465,8 @@ PgColumnarTryGroupAggPath(PlannerInfo *root, RelOptInfo *input_rel, ListCell *lc; int naggs = 0; double dNumGroups; + double timeProduct; + double timeWithNull; Path *cheapest; Cost serialScanCost = 0.0; double serialScanRows = 0.0; @@ -2462,13 +2512,11 @@ PgColumnarTryGroupAggPath(PlannerInfo *root, RelOptInfo *input_rel, /* * estimate_num_groups only sizes the path's row estimate; it is deliberately - * NOT a gate. For an expression key such as date_trunc(...) the planner - * cannot estimate distinctness and returns a count near the input row count - * (here ~8.4M estimated against 48k actual), which would wrongly disable this - * path on exactly the large tables it helps. The unbounded-hash-table guard - * is the execution-time cap on the actual group count - * (pgcolumnar.groupagg_max_groups), which errors with guidance -- see - * pgcolumnar_groupagg_lookup -- rather than silently declining the feature. + * NOT the gate. For an expression key such as date_trunc(...) it cannot see + * distinctness and returns a count near the input row count (~8.4M estimated + * against 48k actual), and for a skewed plain key ANALYZE's own estimator + * can be 68x LOW with fresh statistics (#1330 review: 29,356 estimated for + * 2,000,001). The gate (THE GATE below) uses an upper bound instead. */ groupExprs = groupKeys; dNumGroups = estimate_num_groups(root, groupExprs, @@ -2499,14 +2547,31 @@ PgColumnarTryGroupAggPath(PlannerInfo *root, RelOptInfo *input_rel, * downward: Min() cannot raise an estimate, so a shape we get wrong is priced * no worse than it is today. */ + pgcolumnar_time_bucket_products(root, input_rel, groupExprs, + &timeProduct, &timeWithNull); { - double bound = pgcolumnar_group_estimate_bound(root, input_rel, - groupExprs); + double bound = pgcolumnar_group_estimate_bound(root, groupExprs, + timeProduct); if (bound > 0.0 && bound < dNumGroups) dNumGroups = Max(1.0, bound); } + /* + * THE GATE (#1292 phase 2, the owner's decision: gate now, never error + * later): take the path only when an UPPER BOUND on the group count is + * within the runtime cap. Over the cap the node errors, since its plan is + * fixed by then, so with the path on by default a gate that let a group + * count past the cap would turn a working query into an error. An estimate + * cannot be that gate: a skewed key fools ANALYZE (see above). A bound can, + * and the default then never errors; the cap is reached only if it falls, + * or the table grows past it, after planning. A user who wants more groups + * on this path raises groupagg_max_groups. + */ + if (pgcolumnar_group_count_upper_bound(input_rel, timeWithNull) > + (double) pgcolumnar_groupagg_max_groups) + return; + /* * Pseudoconstant (gating) quals -- e.g. WHERE (SELECT false) -- are one-time * filters, not per-row predicates. The residual ExecQual recheck this path diff --git a/test/groupagg_table_sizing.sh b/test/groupagg_table_sizing.sh index 708a2cb8..feb1aa4a 100755 --- a/test/groupagg_table_sizing.sh +++ b/test/groupagg_table_sizing.sh @@ -61,7 +61,9 @@ psql_run "INSERT INTO gts_expr SELECT '2026-01-01'::timestamptz + (i || ' second FROM generate_series(1,500000) i;" psql_run "ANALYZE gts_expr;" -ON="SET pgcolumnar.enable_group_vectorization=on;" +# The cap is raised so the gate (a tenth of it, #1292 phase 2) admits the 200,000-group +# fixture: this suite measures the hash table's sizing, not the plan choice. +ON="SET pgcolumnar.enable_group_vectorization=on; SET pgcolumnar.groupagg_max_groups=10000000;" BIG="SET work_mem='64MB';" explain_with() { # $1 = extra SET, $2 = query diff --git a/test/native_groupagg.sh b/test/native_groupagg.sh index 6dcf7c40..1dce5679 100755 --- a/test/native_groupagg.sh +++ b/test/native_groupagg.sh @@ -218,26 +218,13 @@ oracle "oracle exact: GROUP BY added column" \ # ---- 7. fallback shapes: not the node, still the right answer --------------- -# The cap bounds the actual group count at execution, not the planner's group -# estimate (unreliable for expression keys): over the cap the node stops with -# guidance rather than building an unbounded hash table. The node is forced here -# (the only grouping path left with hashagg and sort off) so the cap-enforcement -# path is exercised deterministically regardless of how the planner would cost it -# on this small table; the node's real-world plan choice is covered above and by -# the benchmark. The cap is set in-session so the executor reads it directly. -check "plan: node still chosen (no estimate gate)" \ +# Since #1292 phase 2 the planner takes the node only for an estimate of at most a +# tenth of pgcolumnar.groupagg_max_groups, so the runtime cap is reached only when +# the estimate is wrong; section 10 proves that path with a stale-statistics +# fixture. (Before the gate this section forced the node with hashagg and sort off +# under a cap of 100; the gate now declines that plan, which is the point of it.) +check "plan: the node is chosen for a key well under the gate" \ "$(pgc_is_groupvec "SELECT g, count(*) FROM t_col GROUP BY g")" yes -# Capture to a variable before grepping: the query errors on purpose, so psql -# exits non-zero, and under `set -o pipefail` a `psql | grep` pipeline would -# report psql's failure even when grep matched -- flipping this check to a false -# negative. Grepping the captured text keeps psql's exit status out of it. -oc_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -Atc "SET pgcolumnar.enable_group_vectorization=on; - SET enable_hashagg=off; SET enable_sort=off; - SET pgcolumnar.groupagg_max_groups=100; - SELECT g, count(*) FROM t_col GROUP BY g" 2>&1 || true)" -n_err="$(grep -q 'groupagg_max_groups' <<<"$oc_out" && echo yes || echo no)" -check "over-cap stops with a groupagg_max_groups error" "$n_err" yes diff_query "oracle exact: default cap runs the high-cardinality key" \ "SELECT g, count(*), sum(i4) FROM %T GROUP BY g" @@ -549,4 +536,76 @@ check "while a matching-type predicate still gets one" \ check "a non-time expression key gets no bound either" \ "$([ "$(gbest "SELECT upper(host) b, avg(v) FROM gb GROUP BY b")" -gt 100 ] && echo yes || echo no)" "yes" +# ---- 10. on by default, gated on an upper bound (#1292 phase 2) ------------- +# The grouped path is on by default, and the planner takes it only when a TRUE +# UPPER BOUND on the group count is within pgcolumnar.groupagg_max_groups: the +# relation's live row count, or, where every key is a time bucket, the buckets the +# zone maps allow. Over the cap the node errors, since its plan is fixed by then, +# so a gate on an ESTIMATE let a skewed key through: @OffgridwithJD measured on +# #1330 a fresh ANALYZE estimating 29,356 groups for 2,000,001, and the query +# that main answers errored by default. With a bound, the default never errors. +# The database-level setting the sections above use is cleared, so these arms see +# the compiled default. +psql_run "ALTER DATABASE $PGC_DB RESET pgcolumnar.enable_group_vectorization;" +check "the grouped path is on by default" \ + "$(q "SELECT boot_val || '|' || setting FROM pg_settings WHERE name = 'pgcolumnar.enable_group_vectorization';")" "on|on" +# 400,000 rows over two days: 576 five-minute buckets, and k with 200,000 values. +GATE_GEN="SELECT '2026-02-01 00:00:00+00'::timestamptz + g * interval '432 ms', (g % 200000)::int, (g % 1000)::int + FROM generate_series(1, 400000) g" +psql_run "CREATE TABLE gate_t (ts timestamptz, k int, i int) USING pgcolumnar; + INSERT INTO gate_t $GATE_GEN; + CREATE TABLE gate_h (ts timestamptz, k int, i int); + INSERT INTO gate_h $GATE_GEN; + ANALYZE gate_t; ANALYZE gate_h;" +BUCKETQ="SELECT date_bin('5 minutes', ts, '2026-01-01 00:00:00+00') b, count(*), sum(i) FROM gate_t GROUP BY b" +HIGHK="SELECT k, count(*), sum(i) FROM %T GROUP BY k" +# groups_or_error CAP QUERY -> the number of groups the query returns under that cap, +# or its error. The query runs itself: wrapped in SELECT count(*) FROM (...), the +# planner drops the unused aggregates and the node refuses that shape, so the +# wrapper would measure a different plan. +groups_or_error() { + local out + out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -q \ + -c "SET pgcolumnar.groupagg_max_groups = $1" -c "$2" 2>&1)" + if grep -q '^ERROR' <<<"$out"; then grep -m1 '^ERROR' <<<"$out"; else grep -c '' <<<"$out"; fi +} +under_cap() { # under_cap CAP QUERY -> 1 when the grouped node is planned under that cap + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \ + -c "SET pgcolumnar.groupagg_max_groups = $1" -c "EXPLAIN (COSTS OFF) $2" 2>&1 \ + | grep -c 'Columnar Vectorized Group Keys' +} +check "by default a five-minute bucket query takes the grouped path" "$(pgc_is_groupvec "$BUCKETQ")" yes +check "a five-minute bucket query over more rows than the cap still takes the grouped path" \ + "$(under_cap 100000 "$BUCKETQ")" "1" +check "a plain key over more rows than the cap is declined" "$(under_cap 100000 "${HIGHK//%T/gate_t}")" "0" +check "and the declined query answers what heap answers" \ + "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -q \ + -c "SET pgcolumnar.groupagg_max_groups = 100000" \ + -c "SELECT md5(string_agg(x::text, ',' ORDER BY x)) FROM (${HIGHK//%T/gate_t}) x" 2>&1)" \ + "$(q "SELECT md5(string_agg(x::text, ',' ORDER BY x)) FROM (${HIGHK//%T/gate_h}) x;")" +check "raising groupagg_max_groups to 10,000,000 lets the same query take the grouped path" \ + "$(under_cap 10000000 "${HIGHK//%T/gate_t}")" "1" + +# @OffgridwithJD's shape on #1330: one key holds most rows, the rest are unique. +# A fresh ANALYZE samples 30,000 rows and estimates the groups from them, far too +# few (29,356 for 2,000,001 there). Here 95% of 800,000 rows share key 0, so +# 40,001 groups, and under a cap of 30,000 a tenth-of-the-cap gate on the estimate +# (3,000) would take the path and error. The row count, 800,000, declines it. +psql_run "CREATE TABLE gate_sk (k bigint, v int) USING pgcolumnar; + INSERT INTO gate_sk SELECT CASE WHEN g % 20 = 0 THEN g ELSE 0 END, g FROM generate_series(1, 800000) g; + ANALYZE gate_sk;" +SKQ="SELECT k, count(*), sum(v) FROM gate_sk GROUP BY k" +check "premise: ANALYZE estimates the skewed key's 40,001 groups below 3,000" \ + "$(q "SELECT CASE WHEN n_distinct BETWEEN 0 AND 3000 THEN 'below' ELSE n_distinct::text END FROM pg_stats WHERE tablename = 'gate_sk' AND attname = 'k';")" "below" +check "a skewed key whose estimate is far too low is declined under a cap of 30,000, and answers every group" \ + "$(under_cap 30000 "$SKQ")|$(groups_or_error 30000 "$SKQ")" "0|40001" + +# The runtime cap stays as the backstop. With a bound it is reached only when the +# cap falls after planning: a cached plan made under the default runs under 1,000. +rc_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -q \ + -c "PREPARE gk AS ${HIGHK//%T/gate_t}" -c "EXPLAIN (COSTS OFF) EXECUTE gk" -c "EXECUTE gk" \ + -c "SET pgcolumnar.groupagg_max_groups = 1000" -c "EXECUTE gk" 2>&1 || true)" +check "a cached plan run under a lower cap stops at the runtime cap, with its hint" \ + "$(grep -c 'Columnar Vectorized Group Keys' <<<"$rc_out")|$(grep -c 'exceeded pgcolumnar.groupagg_max_groups (1000)' <<<"$rc_out")|$(grep -c 'HINT: Raise pgcolumnar.groupagg_max_groups' <<<"$rc_out")" "1|1|1" + pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 265d8cb1..3aa9a9ed 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -5650,7 +5650,7 @@ every major for this pair, which is the designed third state and not a failure. | `test_the_accumulators_fold_identically_with_the_path_off_and_on` | the float and average accumulators, and the exact ones, byte-identical off versus on | | `test_min_max_keep_the_display_scale_core_keeps` | numeric values equal by value and differing in display scale keep the later one on a tie | | `test_a_lone_negative_zero_keeps_its_sign` | summing `-0.0` prints `-0`, which folding into `+0.0` would lose | -| `test_a_group_count_over_the_cap_stops_rather_than_growing` | over the cap the node errors naming `groupagg_max_groups`, and the default cap runs the high-cardinality key correctly | +| `test_a_key_well_under_the_gate_takes_the_node` | a key well under the gate (#1292) takes the node, and it answers the high-cardinality key as heap does | | `test_an_output_built_on_a_key_falls_back_and_still_answers` | `f(key)` output and a bare `GROUP BY` with no aggregate both fall back, and the answer is still right | | `test_a_non_deterministic_collation_key_falls_back` | the fallback happens and the grouping is case-insensitive; declines under the shell suite's own name where ICU is absent | | `test_sum_real_matches_heap_rather_than_returning_zero` | blocker 1: the node fires, matches heap, and is not zero | @@ -5661,6 +5661,7 @@ every major for this pair, which is the designed third state and not a failure. | `test_the_group_estimate_bound_is_accurate_and_narrow` | the bound is 12 and is an upper bound; an informed estimate is untouched; with no time bound, with only a lower one, and for `date_bin`, the estimate is between the true group count and four times it, from the zone maps (#1292); a lookalike function and a non-time key get no bound | | `test_the_bound_reads_the_zone_maps_in_one_catalog_scan` | planning a bucketed GROUP BY over 50 row groups scans zone_map and its index once in all, not once per group (#1304) | | `test_a_mixed_timestamp_predicate_never_narrows_the_bound` | a cross-type predicate under a non-UTC TimeZone never narrows the estimate below the true group count (#1292); the matching-type one still gets a tight bound | +| `test_on_by_default_and_gated_on_an_upper_bound` | the grouped path is on by default and taken only when an upper bound on the groups (the row count, or the time buckets) is within the cap: a bucket query over more rows than the cap is taken, a plain key over more rows is declined (answering as heap does) until the cap is raised, and a skewed key that fools ANALYZE is declined and answers every group; a cached plan run under a lower cap stops at the runtime cap with its hint (#1292, #1330 review) | ## 72. test_analyze_function.py: statistics collected by reading, not by sampling diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 644c80d2..f6d1e6f3 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -597,4 +597,10 @@ guard_tests 419 # # cluster_tests 525 -> 526 for #1327's test_collation_change.py, one test, by collection # on this branch rebased onto 4c72242 (#1326). -cluster_tests 526 +# +# cluster_tests 526 -> 527 for #1292 phase 2 in test_native_groupagg.py: one test +# renamed (the over-cap test, whose forced plan the gate now declines, became +# test_a_key_well_under_the_gate_takes_the_node) and one added +# (test_on_by_default_and_gated_on_the_estimate), by collection on this branch +# rebased onto c90976d (#1329): 945 collected on main, 946 here. +cluster_tests 527 diff --git a/test/pytest/test_native_groupagg.py b/test/pytest/test_native_groupagg.py index 6973495c..3f908bcf 100644 --- a/test/pytest/test_native_groupagg.py +++ b/test/pytest/test_native_groupagg.py @@ -316,32 +316,17 @@ def test_a_lone_negative_zero_keeps_its_sign(pgc_conn, expect): # ---- shapes the node must decline, still answering correctly ----------------- -def test_a_group_count_over_the_cap_stops_rather_than_growing(pgc_conn, expect): - """The cap bounds the ACTUAL group count at execution, not the planner's estimate. - - The node is forced here -- hashagg and sort off leave it the only grouping path - -- so the cap path is exercised whatever the planner would cost on this fixture. - Its real-world plan choice is covered by the arms above. +def test_a_key_well_under_the_gate_takes_the_node(pgc_conn, expect): + """Since #1292 phase 2 the node is taken only for an estimate of at most a tenth + of pgcolumnar.groupagg_max_groups, so the runtime cap is reached only through a + wrong estimate; test_on_by_default_and_gated_on_the_estimate proves that path. + (Before the gate this forced the node under a cap of 100 with hashagg and sort + off; the gate now declines that plan, which is the point of it.) """ _make_pair(expect, pgc_conn) expect.plan_marker(_plan(pgc_conn, "SELECT g, count(*) FROM t_col GROUP BY g"), - MARKER, name="plan: node still chosen (no estimate gate)") - - # autocommit is on, so the deliberate error below cannot poison what follows. - # Only psycopg.Error is caught: a bug in this file should raise. - _exec(pgc_conn, "SET enable_hashagg=off") - _exec(pgc_conn, "SET enable_sort=off") - _exec(pgc_conn, "SET pgcolumnar.groupagg_max_groups=100") - named = "no" - try: - _rows(pgc_conn, "SELECT g, count(*) FROM t_col GROUP BY g") - except psycopg.Error as exc: - named = "yes" if "groupagg_max_groups" in str(exc) else f"no [{exc}]" - expect.text(named, "yes", "over-cap stops with a groupagg_max_groups error") + MARKER, name="plan: the node is chosen for a key well under the gate") - _exec(pgc_conn, "RESET enable_hashagg") - _exec(pgc_conn, "RESET enable_sort") - _exec(pgc_conn, "RESET pgcolumnar.groupagg_max_groups") # Named at the call rather than through `_oracle`, and without a second node # premise, because the shell suite spends a bare `diff_query` here: the node was # asserted three lines above and asserting it again would publish a premise the @@ -741,3 +726,95 @@ def test_a_mixed_timestamp_predicate_never_narrows_the_bound(pgc_conn, expect): "a mixed timestamp/timestamptz predicate never narrows the estimate below the true count") expect.text("yes" if matching < 5000 else f"no [{matching}]", "yes", "while a matching-type predicate still gets one") + + +# ---- on by default, gated on an upper bound (#1292 phase 2) ----------------- + +def _under_cap(conn, cap, sql): + _exec(conn, f"SET pgcolumnar.groupagg_max_groups = {cap}") + try: + return _plan(conn, sql) + finally: + _exec(conn, "RESET pgcolumnar.groupagg_max_groups") + + +def _groups_or_error(conn, cap, sql): + """The number of groups the query returns under that cap, or its error. The query + runs itself: wrapped in count(*), the planner drops the unused aggregates and the + node refuses that shape, so a wrapper would measure a different plan.""" + _exec(conn, f"SET pgcolumnar.groupagg_max_groups = {cap}") + try: + return str(len(_rows(conn, sql))) + except psycopg.Error as exc: + return f"ERROR: {exc.diag.message_primary}" + finally: + _exec(conn, "RESET pgcolumnar.groupagg_max_groups") + + +def test_on_by_default_and_gated_on_an_upper_bound(pgc_conn, expect): + """The grouped path is on by default, and taken only when a TRUE UPPER BOUND on + the group count (the live row count, or the time buckets the zone maps allow) + is within groupagg_max_groups. A gate on the estimate let a skewed key through + and errored (@OffgridwithJD on #1330); a bound never does. + + Independent of the shell suite's section 10: its own generators; the numbers the + names state are shared. + """ + conn = pgc_conn + expect.text(_one(conn, "SELECT boot_val || '|' || setting FROM pg_settings" + " WHERE name = 'pgcolumnar.enable_group_vectorization'"), "on|on", + "the grouped path is on by default") + gen = ("SELECT '2026-05-01 00:00:00+00'::timestamptz + g * interval '648 ms'," + " ((g * 7) % 200000)::int, (g % 777)::int FROM generate_series(1, 400000) g") + for name, am in (("gate_c", "pgcolumnar"), ("gate_hp", "heap")): + _exec(conn, f"CREATE TABLE {name} (ts timestamptz, k int, i int) USING {am}") + _exec(conn, f"INSERT INTO {name} {gen}") + _exec(conn, f"ANALYZE {name}") + bucket = ("SELECT date_bin('5 minutes', ts, '2026-01-01 00:00:00+00') b, count(*), sum(i)" + " FROM gate_c GROUP BY b") + expect.plan_marker(_plan(conn, bucket), MARKER, + name="by default a five-minute bucket query takes the grouped path") + expect.plan_marker(_under_cap(conn, 100000, bucket), MARKER, + name="a five-minute bucket query over more rows than the cap still takes the grouped path") + high = "SELECT k, count(*), sum(i) FROM {} GROUP BY k" + expect.plan_marker(_under_cap(conn, 100000, high.format("gate_c")), MARKER, absent=True, + name="a plain key over more rows than the cap is declined") + _exec(conn, "SET pgcolumnar.groupagg_max_groups = 100000") + declined = _rows(conn, high.format("gate_c")) + _exec(conn, "RESET pgcolumnar.groupagg_max_groups") + expect.rows(declined, _rows(conn, high.format("gate_hp")), + "and the declined query answers what heap answers") + expect.plan_marker(_under_cap(conn, 10000000, high.format("gate_c")), MARKER, + name="raising groupagg_max_groups to 10,000,000 lets the same query take the grouped path") + + # 95% of 800,000 rows on key 0 and the rest unique: 40,001 groups, which a fresh + # ANALYZE puts far lower. Under a cap of 30,000 the row count declines it. + _exec(conn, "CREATE TABLE gate_skew (k bigint, v int) USING pgcolumnar") + _exec(conn, "INSERT INTO gate_skew SELECT CASE WHEN g % 20 = 7 THEN g * 3 ELSE 0 END, g" + " FROM generate_series(1, 800000) g") + _exec(conn, "ANALYZE gate_skew") + nd = _one(conn, "SELECT n_distinct FROM pg_stats WHERE tablename = 'gate_skew' AND attname = 'k'") + expect.text("below" if nd is not None and 0 <= nd <= 3000 else str(nd), "below", + "premise: ANALYZE estimates the skewed key's 40,001 groups below 3,000") + skq = "SELECT k, count(*), sum(v) FROM gate_skew GROUP BY k" + taken = MARKER in str(_under_cap(conn, 30000, skq)) + expect.text(f"{1 if taken else 0}|{_groups_or_error(conn, 30000, skq)}", "0|40001", + "a skewed key whose estimate is far too low is declined under a cap of 30,000, and answers every group") + + # The runtime cap stays as the backstop: a cached plan made under the default + # cap runs under 1,000. + _exec(conn, f"PREPARE gk AS {high.format('gate_c')}") + planned = MARKER in str(_one(conn, "EXPLAIN (COSTS OFF, FORMAT JSON) EXECUTE gk")) + _rows(conn, "EXECUTE gk") + _exec(conn, "SET pgcolumnar.groupagg_max_groups = 1000") + got = "no error" + try: + _rows(conn, "EXECUTE gk") + except psycopg.Error as exc: + d = exc.diag + got = ("1" if "exceeded pgcolumnar.groupagg_max_groups (1000)" in (d.message_primary or "") else "0") + "|" + \ + ("1" if (d.message_hint or "").startswith("Raise pgcolumnar.groupagg_max_groups") else "0") + _exec(conn, "RESET pgcolumnar.groupagg_max_groups") + _exec(conn, "DEALLOCATE gk") + expect.text(f"{1 if planned else 0}|{got}", "1|1|1", + "a cached plan run under a lower cap stops at the runtime cap, with its hint") diff --git a/test/vector_agg_tlist_shape.sh b/test/vector_agg_tlist_shape.sh index 7d1a80ce..40a6d55c 100755 --- a/test/vector_agg_tlist_shape.sh +++ b/test/vector_agg_tlist_shape.sh @@ -51,9 +51,11 @@ check "premise: the sort column has NULLs, which the aggregates must skip" \ # Vectorized-aggregate node count in the plan. The whole question is which path # ran, so this is asserted per arm rather than assumed. +# The grouped path is pinned off: this suite is about the UNGROUPED fold, and since +# #1292 phase 2 the grouped path is on by default and prints the same marker. vec() { q "SELECT count(*) FROM (SELECT 1) z;" >/dev/null env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -At -c "EXPLAIN (COSTS OFF) $1" 2>/dev/null \ + -d "$PGC_DB" -At -c "SET pgcolumnar.enable_group_vectorization = off" -c "EXPLAIN (COSTS OFF) $1" 2>/dev/null \ | grep -cE 'Columnar Vectorized Aggregates: [1-9]'; } # ansq LABEL QUERY-with-%T : identical answer from heap and columnar.