docs, test: gap-fill, LOCF and interpolation recipes, held to a reference (#1296 phase 1)

CHANGELOG.md

@@ -55,6 +55,16 @@ true until the next version shipped.
   on, the aged rows go, the fresh rows stay, and a table with no retention is never
   touched or named; a failing `expire` does not undo a compaction; and the daemon
   yields `expire`'s lock to an `ACCESS EXCLUSIVE` request within 8 s of a 30 s hold.
+- docs/how-to.md has recipes to fill gaps in a time series, carry the last value
+  forward (LOCF) and interpolate linearly (#1296). These need no feature: phase 0
+  on #1296 measured the gap fill at 0.69 s on 8.75M columnar rows, with the
+  vectorized grouped node still under the LEFT JOIN, and found the `hll` and
+  `tdigest` extensions already read through pgColumnar's scan. The recipes bucket
+  with `date_bin` and the series start as origin, not `date_trunc`: in a session
+  at a half-hour offset (Asia/Kolkata), `date_trunc` made every bucket empty.
+  `test/timeseries_recipes.sh` and an independent `test_timeseries_recipes.py`
+  read each recipe from the document and hold it to a reference computed another
+  way, over a fixture with leading, inner and trailing gaps and an all-NULL hour.
 
 - `test/format_downgrade.sh` and `test_format_downgrade.py` test #1137's promise
   with both real libraries. They build `v1.0-alpha4`, write with it, upgrade to the

docs/how-to.md

@@ -342,6 +342,115 @@ the time zone your sessions use. For a cutoff in your session's time zone, run
 `pgcolumnar.reset_options` has no argument for the retention. To stop expiring,
 set `ttl_interval` to a span no row will reach.
 
+## Fill gaps in a time series
+
+A bucketed query returns no row for a bucket with no data. These three recipes
+return a row for every bucket, then fill the empty ones. They are plain SQL, so
+they need nothing beyond pgColumnar.
+
+The examples use a table `readings (ts timestamptz, v float8)` and one-hour
+buckets over one day. To adapt one, change the table, the column names, the
+range and the width. Keep the origin of `date_bin` equal to the start of the
+series, so every bucket the aggregate makes is a bucket the series has.
+
+`date_bin` with an explicit origin does not depend on the session's time zone.
+`date_trunc('hour', ts)` does: in a zone with a half-hour offset, its buckets
+start at half past the hour and match no bucket in the series.
+
+### Fill empty buckets
+
+Join the bucketed aggregate to a series of every bucket. An empty bucket
+returns `NULL`.
+
+```sql
+SELECT b.bucket, a.avg_v
+  FROM generate_series(timestamptz '2026-10-01 00:00+00',
+                       timestamptz '2026-10-01 23:00+00',
+                       interval '1 hour') AS b(bucket)
+  LEFT JOIN (SELECT date_bin(interval '1 hour', ts, timestamptz '2026-10-01 00:00+00') AS bucket,
+                    avg(v) AS avg_v
+               FROM readings
+              GROUP BY 1) AS a USING (bucket)
+ ORDER BY b.bucket;
+```
+
+With `pgcolumnar.enable_group_vectorization` on, the aggregate under the join
+runs as the vectorized grouped node.
+
+### Carry the last value forward
+
+Last observation carried forward (LOCF) fills an empty bucket with the value of
+the latest bucket before it that has one. A running count of non-null values
+puts each empty bucket in the same group as the last known value, and
+`first_value` reads that value.
+
+```sql
+WITH filled AS (
+SELECT b.bucket, a.avg_v
+  FROM generate_series(timestamptz '2026-10-01 00:00+00',
+                       timestamptz '2026-10-01 23:00+00',
+                       interval '1 hour') AS b(bucket)
+  LEFT JOIN (SELECT date_bin(interval '1 hour', ts, timestamptz '2026-10-01 00:00+00') AS bucket,
+                    avg(v) AS avg_v
+               FROM readings
+              GROUP BY 1) AS a USING (bucket)
+), grouped AS (
+  SELECT bucket, avg_v, count(avg_v) OVER (ORDER BY bucket) AS grp
+    FROM filled
+)
+SELECT bucket, avg_v,
+       first_value(avg_v) OVER (PARTITION BY grp ORDER BY bucket) AS avg_v_locf
+  FROM grouped
+ ORDER BY bucket;
+```
+
+A bucket before the first value stays `NULL`, because there is nothing to
+carry. A bucket whose rows all have `v` null has an average of `NULL`, so it
+counts as empty and is filled.
+
+### Interpolate between known values
+
+Linear interpolation fills an empty bucket on the straight line between the
+known buckets on either side. A running count in each direction finds both
+neighbours.
+
+```sql
+WITH filled AS (
+SELECT b.bucket, a.avg_v
+  FROM generate_series(timestamptz '2026-10-01 00:00+00',
+                       timestamptz '2026-10-01 23:00+00',
+                       interval '1 hour') AS b(bucket)
+  LEFT JOIN (SELECT date_bin(interval '1 hour', ts, timestamptz '2026-10-01 00:00+00') AS bucket,
+                    avg(v) AS avg_v
+               FROM readings
+              GROUP BY 1) AS a USING (bucket)
+), marks AS (
+  SELECT bucket, avg_v,
+         count(avg_v) OVER (ORDER BY bucket) AS g_prev,
+         count(avg_v) OVER (ORDER BY bucket DESC) AS g_next
+    FROM filled
+), ends AS (
+  SELECT bucket, avg_v,
+         first_value(bucket) OVER (PARTITION BY g_prev ORDER BY bucket) AS prev_t,
+         first_value(avg_v)  OVER (PARTITION BY g_prev ORDER BY bucket) AS prev_v,
+         first_value(bucket) OVER (PARTITION BY g_next ORDER BY bucket DESC) AS next_t,
+         first_value(avg_v)  OVER (PARTITION BY g_next ORDER BY bucket DESC) AS next_v
+    FROM marks
+)
+SELECT bucket,
+       CASE WHEN avg_v IS NOT NULL THEN avg_v
+            WHEN prev_v IS NULL OR next_v IS NULL THEN NULL
+            ELSE prev_v + (next_v - prev_v)
+                 * extract(epoch FROM bucket - prev_t)
+                 / extract(epoch FROM next_t - prev_t)
+       END AS avg_v_interp
+  FROM ends
+ ORDER BY bucket;
+```
+
+A bucket before the first value or after the last one stays `NULL`, because it
+has a neighbour on one side only.
+
 ## Create time partitions ahead
 
 `pgcolumnar.partition_create_ahead` keeps a time-partitioned table ready for

test/check_ledger.tsv

@@ -2032,6 +2032,16 @@ tid_range_scan	tid_range_scan	premise: the subquery cursor plans a TID range sca
 tid_range_scan	tid_range_scan	premise: the table has many row groups, so skipping can show	15;16;17;18;19	never	-
 tid_range_scan	tid_range_scan	premise: the table holds every row	15;16;17;18;19	never	-
 tid_range_scan	tid_range_scan	rows inserted earlier in the same transaction are in the range, as the filter sees them	15;16;17;18;19	never	-
+timeseries_recipes	timeseries_recipes	premise: the readings table is columnar	15;16;17;18;19	never	-
+timeseries_recipes	timeseries_recipes	premise: the reference has 24 hours, 6 of them empty and one all-NULL	15;16;17;18;19	never	-
+timeseries_recipes	timeseries_recipes	the LOCF recipe carries the last value forward, and leaves a leading gap NULL	15;16;17;18;19	2026-10-13	the how-to's 'Carry the last value forward' heading renamed, so the recipe is no longer found;the how-to's LOCF recipe carries with lag(), one bucket only, instead of first_value over the running-count group;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
+timeseries_recipes	timeseries_recipes	the gap-fill recipe gives the same buckets in a session at a half-hour offset	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found;the how-to's recipes bucket with date_trunc('hour', ts) instead of date_bin with the series start as origin;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
+timeseries_recipes	timeseries_recipes	the gap-fill recipe returns every bucket, empty ones as NULL	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
+timeseries_recipes	timeseries_recipes	the gap-fill recipe's aggregate runs as the vectorized grouped node when it is on	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found;the how-to's recipes aggregate avg(v ORDER BY v), which the vectorized grouped node does not take
+timeseries_recipes	timeseries_recipes	the how-to gives the LOCF recipe	15;16;17;18;19	2026-10-13	the how-to's 'Carry the last value forward' heading renamed, so the recipe is no longer found
+timeseries_recipes	timeseries_recipes	the how-to gives the gap-fill recipe	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found
+timeseries_recipes	timeseries_recipes	the how-to gives the interpolation recipe	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found
+timeseries_recipes	timeseries_recipes	the interpolation recipe fills a gap linearly, and leaves both ends NULL	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found;the how-to's interpolation recipe uses prev_v in place of next_v, so every gap is filled flat;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
 validity_elision	validity_elision	a chunk claiming no bitmap while it holds fewer values than rows is refused	15;16;17;18;19	never	-
 validity_elision	validity_elision	a column with no nulls stores no validity bitmap	15;16;17;18;19	2026-09-18	writer: flush_one_column's presentCount == rowCount forced false, so the bitmap is written even when the chunk holds no null
 validity_elision	validity_elision	a null-free column elides its bitmap beside a null-bearing one in the same row group	15;16;17;18;19	never	-

test/check_ledger_budget.txt

@@ -1428,4 +1428,19 @@ suites_not_covered 247
 #     never         1686 + 12 = 1698
 #     dated          317 + 44 =  361
 #     1698 + 361 == 2059
-checks_never_observed_red 1698
+#
+# #1296 phase 1 (timeseries_recipes, PR #1311): 1698 -> 1700. 10 checks from
+# five-major runs on this tree, so the suite is covered and suites_not_covered
+# stays 247. Eight are dated by eight mutations of the recipes in docs/how-to.md,
+# each merged one log at a time with --mutation and --target: date_trunc
+# buckets, an inner join, LOCF by lag, interpolation toward the wrong
+# neighbour, an ordered aggregate, and each of the three recipe headings
+# renamed. The three 'the how-to gives the ... recipe' checks were premises in
+# the first draft; editing the document reddens them, so by #1236 they are arms
+# (OffgridwithJD's review of #1311). The two still never are fixture premises.
+#
+#     ledger rows   2059 + 10 = 2069
+#     never         1698 +  2 = 1700
+#     dated          361 +  8 =  369
+#     1700 + 369 == 2069
+checks_never_observed_red 1700

test/pytest/TESTS.md

@@ -143,6 +143,7 @@ behaviour, the source of that number is named.
 - [95. test_docs_how_to_sections.py: the pg_partman how-to keeps its own heading](#95-test_docs_how_to_sectionspy-the-pg_partman-how-to-keeps-its-own-heading)
 - [96. test_autovacuum_expire.py: the daemon expires retention tables only when told to](#96-test_autovacuum_expirepy-the-daemon-expires-retention-tables-only-when-told-to)
 - [97. test_rollup.py: incremental rollups fold appended row groups, and equal a full recompute](#97-test_rolluppy-incremental-rollups-fold-appended-row-groups-and-equal-a-full-recompute)
+- [98. test_timeseries_recipes.py: the gap-fill, LOCF and interpolation recipes answer what they claim](#98-test_timeseries_recipespy-the-gap-fill-locf-and-interpolation-recipes-answer-what-they-claim)
 
 ## 1. How to read a test in here
 
@@ -6619,3 +6620,22 @@ Removal proofs, each red on its arm: no `OFFSET 0` (the cost arm), no isolation
 | test | what it holds |
 | --- | --- |
 | `test_rollup_phase_one` | the rollup equals a full recompute after the first refresh, appends with late rows, a bulk append, and a rebase; a refresh with nothing new folds nothing; rows committed during a refresh are folded by the next, once; two concurrent refreshes fold each row once; zones agree; a refresh after a small append reads under a fifth of the source; a rewritten source is refused until `rebase_rollup`; a delete is refused (phase 1); a heap rollup behaves the same; REPEATABLE READ is refused; an owner who is not a superuser creates and refreshes a rollup, and another role sees none of it; bad definitions are refused with `22023` |
+
+## 98. test_timeseries_recipes.py: the gap-fill, LOCF and interpolation recipes answer what they claim
+
+Port of `timeseries_recipes.sh` (#1296). Phase 0 on #1296 found that core SQL expresses gap filling, last observation carried forward and linear interpolation fast enough on columnar, so the deliverable is three recipes in `docs/how-to.md`, and this file holds each one to a reference.
+
+THE RECIPES ARE READ FROM THE DOCUMENT, as the shell suite reads them: the first `sql` block under each heading, run as written against the table the document names. Everything else is this file's own. The rows are generated in Python, and the reference is computed in Python from those same rows, so it shares nothing with the shell suite's plpgsql reference. `compare_to_bash` grades the pair 10 of 10.
+
+THE FIXTURE holds an empty first hour, a run of empty hours, an hour with some NULL values, an hour of only NULL values and two empty last hours. Values grow as the square of the hour, so a line through the wrong neighbours lands on the wrong value.
+
+THE HALF-HOUR ZONE ARM is why the recipes bucket with `date_bin` and an explicit origin. `date_trunc` truncates in the session's zone, so in Asia/Kolkata its buckets start at :30 UTC. With `date_trunc` in the document, this arm read all 24 buckets as NULL.
+
+Red first: on main the how-to has no recipes, so the premise reads 0 and the comparisons get nothing.
+
+### Every test
+
+| test | what it holds |
+| --- | --- |
+| `test_the_recipes_are_in_the_how_to` | each of the three headings has a `sql` block naming `readings` |
+| `test_each_recipe_matches_the_reference` | the table is columnar; the gap fill, LOCF and interpolation match the Python reference for all 24 hours; the gap fill gives the same buckets in an Asia/Kolkata session; its aggregate runs as the vectorized grouped node when that is on |

test/pytest/expected_tests.txt

@@ -574,4 +574,7 @@ guard_tests 415
 #
 # cluster_tests 510 -> 511 for #1294's test_rollup.py, one test, collected over the
 # cluster files on this branch rebased onto b6656ac (#1307).
-cluster_tests 511
+#
+# cluster_tests 511 -> 513 for #1296's test_timeseries_recipes.py, two tests, by collection
+# on this branch rebased onto 024e737.
+cluster_tests 513

test/pytest/test_compare_to_bash.py

@@ -136,6 +136,7 @@
             "storage_row_repair",
             "temporal",
             "tid_range_scan",
+            "timeseries_recipes",
             "ttl_expire",
             "validity_elision",
             "zonemap_boundaries"]