feat: partition skew, per-table autovacuum tuning, pg_stat_io verdict in why; fork-safe release #4
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| --- | ||
| id: autovacuum_table_tuning | ||
| severity: info | ||
| critical_when: "" | ||
| dimension: storage | ||
| object: relation | ||
| scope: workload | ||
| requires: [] | ||
| thresholds: [] | ||
| related: [autovacuum_starved, table_bloat, txid_wraparound] | ||
| --- | ||
|
|
||
| # autovacuum_table_tuning | ||
|
|
||
| **Severity:** info · **Dimension:** storage · **Object identity:** `schema.table` (see [configuration](../configuration.md)) · **Requires:** — | ||
|
|
||
| ## What pgbot observed | ||
|
|
||
| A table with **≥ 1,000,000** live rows (`avTuneMinRows`), write activity (dead | ||
| tuples or updates on record), autovacuum enabled, and **no per-table | ||
| `autovacuum_vacuum_scale_factor`** override, while the global scale factor is | ||
| **≥ 0.1** (`avTuneMinScale`; the default is 0.2). pgbot reports the trigger the | ||
|
Comment on lines
+19
to
+22
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 12 \
'autovacuumTableTuning|autovacuum.*off|Params.*autovacuum|setting.*autovacuum' \
internal/findingsRepository: PyModel/pgbot Length of output: 39544 🏁 Script executed: set -euo pipefail
sed -n '803,860p' internal/findings/findings.go
rg -n -A12 -B4 'func setting(Float|Param)|func Compute\(' internal/findings/findings.go
sed -n '19,26p' docs/findings/autovacuum_table_tuning.mdRepository: PyModel/pgbot Length of output: 5579 Skip table tuning when global autovacuum is disabled. When 🤖 Prompt for AI Agents |
||
| table currently waits for — `autovacuum_vacuum_threshold + scale × n_live_tup` — | ||
| and the trigger a per-table override of scale 0.02 / threshold 1000 | ||
| (`avTuneSuggestedScale`, `avTuneSuggestedThres`) would give. Up to ten tables | ||
| are listed, largest first. | ||
|
|
||
| ## Why it matters | ||
|
|
||
| `autovacuum_vacuum_scale_factor` is a fraction of the table, so the same 20% | ||
| that is fine for a 10k-row lookup table means a 50M-row `orders` table | ||
| accumulates 10M dead rows before autovacuum even starts. Until then those rows | ||
| bloat the heap and every index, slow every scan, and let the table's | ||
| transaction-id age climb. When the vacuum finally runs it is a big one that | ||
| holds a worker for a long time. The documented remedy is a per-table override | ||
| that fires on a small fraction plus a fixed threshold — set on the relation, so | ||
| every other table keeps the default and the global cost budget is not spent on | ||
| tiny ones. | ||
|
|
||
| ## How to verify it yourself | ||
|
|
||
| ```sql | ||
| -- Current effective trigger per large table (global settings + reloptions). | ||
| SELECT s.schemaname, s.relname, s.n_live_tup, s.n_dead_tup, | ||
| coalesce((SELECT option_value::float | ||
| FROM pg_options_to_table(c.reloptions) | ||
| WHERE option_name = 'autovacuum_vacuum_scale_factor'), | ||
| current_setting('autovacuum_vacuum_scale_factor')::float) AS scale, | ||
| coalesce((SELECT option_value::int | ||
| FROM pg_options_to_table(c.reloptions) | ||
| WHERE option_name = 'autovacuum_vacuum_threshold'), | ||
| current_setting('autovacuum_vacuum_threshold')::int) AS threshold | ||
| FROM pg_stat_user_tables s | ||
| JOIN pg_class c ON c.oid = s.relid | ||
| WHERE s.n_live_tup >= 1000000 | ||
| ORDER BY s.n_live_tup DESC; | ||
| ``` | ||
|
|
||
| The trigger is `threshold + scale × n_live_tup`. | ||
|
|
||
| ## How to fix it | ||
|
|
||
| ```sql | ||
| ALTER TABLE public.orders SET ( | ||
| autovacuum_vacuum_scale_factor = 0.02, | ||
| autovacuum_vacuum_threshold = 1000, | ||
| autovacuum_analyze_scale_factor = 0.01, | ||
| autovacuum_analyze_threshold = 500 | ||
| ); | ||
| ``` | ||
|
|
||
| Takes effect at the next autovacuum cycle; no restart, and no rewrite (a brief | ||
| `SHARE UPDATE EXCLUSIVE` lock). Derive the numbers from the table: a queue-like | ||
| table with a few thousand live rows and constant churn wants a threshold-driven | ||
| trigger; a billion-row fact table wants an even smaller scale factor. More | ||
| frequent vacuums on big tables need cost budget — watch | ||
| [autovacuum_saturated](autovacuum_saturated.md) and raise | ||
| `autovacuum_vacuum_cost_limit` if workers fall behind. | ||
|
|
||
| Rollback: `ALTER TABLE public.orders RESET (autovacuum_vacuum_scale_factor, | ||
| autovacuum_vacuum_threshold, autovacuum_analyze_scale_factor, | ||
| autovacuum_analyze_threshold);` | ||
|
|
||
| ## When to ignore it | ||
|
|
||
| - Append-only tables with occasional deletes — pgbot already skips tables with no | ||
| dead tuples and no updates, but a rare bulk delete can trip it. | ||
| - You lowered the global scale factor deliberately (below 0.1 the finding stays quiet). | ||
|
|
||
| ```toml | ||
| [[ignore]] | ||
| finding = "autovacuum_table_tuning" | ||
| object = "public.audit_log" | ||
| reason = "append-only; monthly partition drop handles retention" | ||
| expires = "2027-01-01" | ||
| ``` | ||
|
|
||
| ## What pgbot cannot see | ||
|
|
||
| - The write *rate*: it sees dead tuples and cumulative updates, not how fast they | ||
| arrive, so it cannot say how long the table waits between vacuums. | ||
| - Whether a manual `VACUUM` schedule already covers the table. | ||
| - `autovacuum_vacuum_insert_*` (PG13+) for insert-only tables — a separate trigger | ||
| this finding does not model. | ||
|
|
||
| ## Related | ||
|
|
||
| - [autovacuum_starved](autovacuum_starved.md) — the trigger was reached and | ||
| autovacuum still didn't run; this finding is the trigger being too far away. | ||
| - [table_bloat](table_bloat.md) — what accumulates while waiting for the trigger. | ||
| - [txid_wraparound](txid_wraparound.md) — the eventual cost of vacuums that come too late. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| --- | ||
| id: partition_skew | ||
| severity: info | ||
| critical_when: "" | ||
| dimension: throughput | ||
| object: relation | ||
| scope: workload | ||
| requires: [] | ||
| thresholds: [] | ||
| related: [partition_seq_scan_heavy, autovacuum_table_tuning] | ||
| --- | ||
|
|
||
| # partition_skew | ||
|
|
||
| **Severity:** info · **Dimension:** throughput · **Object identity:** `schema.table` (the partitioned parent; see [configuration](../configuration.md)) · **Requires:** — | ||
|
|
||
| ## What pgbot observed | ||
|
|
||
| For a partitioned table with at least **4** leaf partitions | ||
| (`partitionSkewMinParts`), the hottest leaf takes **≥ 4×** the per-partition | ||
| average scan count (`partitionSkewFactor`, over at least 1,000 total scans), or | ||
| the largest leaf holds ≥ 4× the average row count (over at least 100,000 rows). | ||
| Scan counts are `seq_scan + idx_scan` from `pg_stat_user_tables`, rolled up by | ||
| climbing `pg_inherits` to the root. The finding is suppressed on a cold stats | ||
| window. | ||
|
|
||
| ## Why it matters | ||
|
|
||
| Partitioning spreads maintenance and scans only as far as the key spreads the | ||
| data. One leaf carrying most of the rows or reads is, for every purpose that | ||
| matters — vacuum duration, index build time, scan cost, lock scope — an | ||
| unpartitioned table with extra planning overhead. It is also the earliest | ||
| visible form of the hot-shard problem: the same key would put the same tenant or | ||
| value on one shard if the table were ever distributed, and no number of routers | ||
| or shards fixes a key that doesn't spread. | ||
|
|
||
| ## How to verify it yourself | ||
|
|
||
| ```sql | ||
| -- Per-leaf scans and rows for one partitioned parent, hottest first. | ||
| SELECT c.relname AS partition, | ||
| s.seq_scan + coalesce(s.idx_scan, 0) AS scans, | ||
| s.n_live_tup AS rows | ||
| FROM pg_inherits i | ||
| JOIN pg_class c ON c.oid = i.inhrelid | ||
| JOIN pg_stat_user_tables s ON s.relid = c.oid | ||
| WHERE i.inhparent = '<parent_table>'::regclass | ||
|
Comment on lines
+44
to
+47
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Make the verification query traverse descendant partitions. This query only reads direct children of 🤖 Prompt for AI Agents |
||
| ORDER BY scans DESC; | ||
| ``` | ||
|
|
||
| ## How to fix it | ||
|
|
||
| - **Time-range key, hottest = newest partition:** expected. Ignore it (below). | ||
| - **List key with one dominant value** (a big tenant, a common status): sub-partition | ||
| that value (`PARTITION OF … FOR VALUES IN ('big_tenant') PARTITION BY HASH (id)`), | ||
| or move it to its own table with the same schema. | ||
| - **Hash key with low cardinality:** re-key on something with more distinct | ||
| values, e.g. `(tenant_id, id)` hashed together, at the next rebuild. | ||
| - **Accept it:** give the hot leaf its own autovacuum settings | ||
| ([autovacuum_table_tuning](autovacuum_table_tuning.md)) and confirm its indexes | ||
| are the ones the hot queries need. | ||
|
|
||
| Re-partitioning is a table rewrite — plan it as a migration with | ||
| `CREATE … CONCURRENTLY` indexes and a cut-over, not an `ALTER`. | ||
|
|
||
| ## When to ignore it | ||
|
|
||
| Time-based partitioning where the newest partition is hot by design, or an | ||
| archive layout where old partitions are deliberately cold. | ||
|
|
||
| ```toml | ||
| [[ignore]] | ||
| finding = "partition_skew" | ||
| object = "public.events" | ||
| reason = "monthly range partitions; current month is hot by design" | ||
| expires = "2027-01-01" | ||
| ``` | ||
|
|
||
| ## What pgbot cannot see | ||
|
|
||
| - Counters are cumulative since the stats reset: a partition attached last week | ||
| looks cold next to one attached last year. | ||
| - Which *value* is hot — only which leaf. Map the leaf to its bound with | ||
| `pg_get_expr(relpartbound, oid)`. | ||
| - Query-level routing: whether hot queries prune to one leaf or scan all of them | ||
| ([partition_seq_scan_heavy](partition_seq_scan_heavy.md) covers the latter). | ||
|
|
||
| ## Related | ||
|
|
||
| - [partition_seq_scan_heavy](partition_seq_scan_heavy.md) — the parent scanned | ||
| end-to-end; the other way partitioning fails to pay off. | ||
| - [autovacuum_table_tuning](autovacuum_table_tuning.md) — the hot leaf is exactly | ||
| the relation that needs its own vacuum trigger. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: PyModel/pgbot
Length of output: 5433
Use the releasing repository for the Docker smoke test.
The GoReleaser step publishes to
ghcr.io/${{ github.repository }}, but.github/workflows/release.yml:163runsghcr.io/pgrundev/pgbot:"$VERSION". In fork releases, the smoke test can validate the upstream image instead of the image that this workflow published. SetIMAGE_REPOfor the smoke step or job and run"$IMAGE_REPO:$VERSION".🤖 Prompt for AI Agents