docs, test: gap-fill, LOCF and interpolation recipes, held to a reference (#1296 phase 1)

[jdatcmd]

Phase 1 of #1296. Phase 0's numbers and the decision are on the issue: recipes and existing extensions cover all three, and nothing is built for 1.0. Phase 2, the hll and tdigest compatibility arms, comes as its own PR.

What users get

A new how-to section, "Fill gaps in a time series", with three plain-SQL recipes over readings (ts, v) in one-hour buckets:

• Fill empty buckets: a series LEFT JOIN the bucketed aggregate.
• Carry the last value forward: a running count of non-null values, then first_value in each group.
• Interpolate between known values: running counts in both directions find both neighbours, then a straight line between them.

The buckets come from date_bin, with the series start as origin, not date_trunc. date_trunc truncates in the session's time zone. In an Asia/Kolkata session (+5:30), its buckets start at :30 UTC and match nothing in the series, so with date_trunc in the document, every bucket came back empty. The docs say why.

Tests: timeseries_recipes.sh and an independent test_timeseries_recipes.py, 10 checks each

• The recipes are read from docs/how-to.md: the first sql block under each heading, run as written. An edited recipe is the recipe tested.
• The references are computed differently in each twin: a plpgsql loop over the raw rows in the shell suite, and Python over the inserted rows in the twin. Neither shares a window function or a join with the recipes.
• The fixture:
  • an empty first hour, a run of empty hours, an hour with some NULLs, an hour of only NULLs, and two empty last hours;
  • values that grow as the square of the hour, so a line through the wrong neighbours gives the wrong value;
  • every value a multiple of 1/16, so comparisons are exact.
• The checks:
  • each recipe matches the reference for all 24 hours;
  • the gap fill gives the same buckets in a session at a half-hour offset;
  • with enable_group_vectorization on, its aggregate runs as the vectorized grouped node.

Red first. On main, the shell suite (PG17) and the twin (PG16) find no recipes: the premises read 0 and the comparisons get nothing.

Mutations, each run on 15–19 and merged into the ledger one log at a time:

mutation of the recipes	check that turned red
date_trunc buckets	the half-hour zone check (all 24 buckets NULL)
JOIN instead of LEFT JOIN	the fill, LOCF and interpolation checks, and the zone check
LOCF by lag()	the LOCF check (03 carried, 04 NULL)
interpolation toward prev_v	the interpolation check (a flat fill)
avg(v ORDER BY v)	the vectorized-node check
the heading renamed	the gap-fill premise and its three checks

Ledger: 10 rows, 6 dated, and the 4 still never are premises. Census 1693 → 1697, rows 1943 → 1953, suites_not_covered stays 247. compare_to_bash grades the pair 10 of 10. docs_style is green.

Registries: TESTS.md section 93, cluster_tests 500 → 502 by collection. #1307 also takes section 93; whichever merges second renumbers.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, this is ready for review. It's rebased onto d7c9951 (#1312), with head 9e27071.

• The rebase: only the budget file conflicted. The merged ledger is main's 1945 rows plus this PR's 10, verified row by row. Census 1694 → 1698, rows 1945 → 1955. The budget text says the mutation runs were of this PR on its first base, 3deb6af.
• Re-run on the rebased tree: timeseries_recipes is 10/10 on each of 15–19.
• Counts: cluster_tests collected at 502, unchanged; the ledger guards give 119 passed.

What to attack: whether the plpgsql and Python references really are independent of the recipes, and whether the half-hour zone check could pass for the wrong reason. It was red with date_trunc in the document.

[OffgridwithJD]

The recipes, the references and the mutation table are solid. The date_bin-not-date_trunc point with the Kolkata measurement is exactly the trap a user hits. One #1236 point blocks, and it's the same one as #1313:

A dated premise. The ledger dates premise: the how-to gives the gap-fill recipe with the heading-renamed mutation. Editing the document is a repository change, so this check is an arm. Its two siblings, premise: the how-to gives the LOCF recipe and premise: the how-to gives the interpolation recipe, read their own headings the same way. Their mutation wasn't run, but renaming either heading would redden them just the same. So I'd drop the premise: prefix from all three in both twins and re-run, with the logs carrying the new names.

The other two premises, the readings table being columnar and the reference's 24 hours, are fixture facts no repository change touches. They're right as premises.

Everything else checked out here: 10 ledger rows, 6 dated, every dated row matching your table, and the arithmetic 1693 → 1697.

🤖 Generated with Claude Code

[jdatcmd]

Taken: you're right, as on #1313. Head is now f349004, rebased onto 7e6fd48 (#1318).

The three checks are arms now. the how-to gives the gap-fill / LOCF / interpolation recipe dropped the premise: prefix in both twins, with a comment saying why.

Fresh runs, because the old logs carry the old names. A clean run plus eight mutations, each on all five majors:

mutation	check that turned red
the six from before	as in the PR description
'Fill empty buckets' heading renamed	gap-fill recipe found, and its three dependent checks
'Carry the last value forward' renamed (new)	the how-to gives the LOCF recipe, and the LOCF comparison
'Interpolate between known values' renamed (new)	the how-to gives the interpolation recipe, and the interpolation comparison

• Ledger: I dropped this suite's rows on the rebase and re-merged from the fresh logs. 10 rows, 8 dated; the 2 still never are the fixture premises you named. Census 1705 → 1707, rows 1984 → 1994.
• History: I rebuilt it as four commits (red suite, docs, twin plus registries, ledger), because a commit message had come to describe a ledger change it no longer held.
• Rebase: onto test, docs: hll and tdigest over columnar tables, the compatibility statement (#1296 phase 2) #1315 and docs: restore the pg_partman how-to heading my #1315 edit removed #1318. TESTS.md section 96, cluster_tests 505 by collection. The docs headings, diffed against main, show only additions.
• Re-run on this exact tree: timeseries_recipes 10/10 on PG17; docs_style green; ledger and docs guards 120 passed.

[OffgridwithJD]

Approved at f349004. I re-checked it row by row against 7e6fd48:

• 1984 → 1994 rows, all 10 new.
• The three the how-to gives the ... recipe checks are arms now. Each is dated by its own heading-rename mutation, as your table says.
• The 2 never rows are the fixture premises (a columnar table, and the reference's 24 hours). No premise is dated.
• The census is 1707, matching the budget line.

🤖 Generated with Claude Code

[jdatcmd]

Thanks for the approval at f349004. #1314 and #1313 merged first, so I've rebased onto 953c192. Head is now 1d5f0f9; please re-approve.

• check_ledger.tsv: the +/- lines are identical to the 10 you approved, compared sorted with diff.
• The budget, restated from main: 1684 → 1686, rows 1994, dated 300 + 8 = 308.
• cluster_tests: 504 → 506 by collection, after feat: expire records each row group it retires, for rollups that keep history (#1294) #1314's test.
• Guards: ledger and docs, 119 passed.

No other file's patch changed.

[jdatcmd]

#1320 merged before your re-approval came in, so I've rebased once more, onto c0ee868. Head is now 74d2c94, which replaces 1d5f0f9. check_ledger.tsv +/- lines are identical to the 10 you approved; the budget is restated as 1683 → 1685, rows 1994, dated 301 + 8 = 309; test_mutation_ledger.py 39 passed. Nothing else changed.

[OffgridwithJD]

Re-approved at 74d2c94, after checking the tree myself:

• Rebased onto c0ee868 (4 commits). The file set equals f349004's, and every file's +/- lines equal the ones I approved, compared sorted, apart from the budget and expected_tests restatements.
• Census, computed from the ledger: 1994 rows, 1685 never, against main's 1984/1683. The budget says 1685.
• cluster_tests: 506 collected over the cluster files on this tree, and the file says 506.
• test_mutation_ledger.py 39 passed; docs_style PASSED.

[jdatcmd]

Thanks for the approval at 74d2c94. #1307 merged first, so I've rebased onto b6656ac. Head is now 39d4b9c; please re-approve.

• check_ledger.tsv: +/- lines identical to the 10 you approved.
• The budget, restated: 1686 → 1688, rows 2003 → 2013, dated 317 + 8 = 325.
• TESTS.md: section 97, after feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290) #1307's 96. cluster_tests 510 → 512 by collection.
• Guards: ledger and docs, 119 passed.

No other file's patch changed.

[jdatcmd]

#1319 merged before you got to 39d4b9c, so I've rebased again, onto 024e737. Head is now 2fbc40c, which replaces 39d4b9c.

• check_ledger.tsv: +/- lines identical to the 10 you approved.
• The budget: 1698 → 1700, rows 2059 → 2069, dated 361 + 8 = 369.
• TESTS.md: section 98, after feat: incremental rollups, phase 1 (append-only) (#1294) #1319's 97. cluster_tests 511 → 513 by collection.
• Guards: 119 passed.

No other file's patch changed.

[OffgridwithJD]

Re-approved at 2fbc40c, after checking the tree myself:

• Rebased onto 024e737. Every file's +/- lines equal the ones I approved at 74d2c94, compared sorted, apart from three restatements: the budget, expected_tests, and TESTS.md. TESTS.md is identical except for the section number, now 98.
• Census, computed from the ledger: 2069 rows and 1700 never, which matches the budget.
• cluster_tests: 513 collected on this tree, and the file says 513.
• test_mutation_ledger.py 39 passed; docs_style PASSED.

Note that #1322 also takes TESTS.md section 98; whichever merges second renumbers.