docs, test: the latest-reading-per-device recipe, run from the document (#1293)

CHANGELOG.md

@@ -39,6 +39,18 @@ true until the next version shipped.
   parity again after the newest and oldest rows are deleted. The first draft of
   the fixture overflowed `int`, its `INSERT` failed without a word, and every arm
   measured an empty table. A row-count premise now comes first.
+- `docs/how-to.md` has a recipe for the latest reading per device (#1293), and
+  `test/latest_reading.sh` with its pytest twin runs the recipe's SQL straight from
+  the document. Each documented form must return the rows `DISTINCT ON` returns;
+  with the documented btree the `LATERAL` form must read under a tenth of the
+  table; and without it, the trap the section warns about must hold: the table is
+  read about once per device. Measured on #1293 with 10 million rows and 1,000
+  devices: 45 ms with the btree, 12.6 minutes without it, and 5.6 seconds for
+  `DISTINCT ON`. Editing the document's recipe to order ascending reddens the
+  parity arm on every major. The first red run, before the section existed, also
+  showed a vacuous arm: "under a tenth of the table" passed on empty SQL, because
+  a query that reads nothing reads under a tenth of anything. It now requires at
+  least one block read.
 
 ### Changed
 

docs/how-to.md

@@ -207,6 +207,48 @@ serves poorly. Confirm the plan shows `Columnar Projection`. Keep the column lis
 minimal, because a projection that misses one queried column is never used. See
 Projections in the administration guide for detail.
 
+## Find the latest reading per device
+
+"What is the latest reading for each device?" is the usual time-series question.
+`max(ts)` over a whole table is answered from zone maps and reads almost no data.
+Per device, choose one of two forms.
+
+To answer it fast for many devices, add a btree on the device and the timestamp:
+
+```sql
+CREATE INDEX readings_device_ts ON readings (device, ts);
+```
+
+When you need only the latest timestamp, group by device. This reads the table
+once:
+
+```sql
+SELECT device, max(ts) FROM readings GROUP BY device ORDER BY device;
+```
+
+When you need the whole latest row, look each device up through the index. The
+`devices` table lists the devices, one row each:
+
+```sql
+SELECT d.device, x.ts
+FROM devices d
+CROSS JOIN LATERAL (
+    SELECT ts FROM readings r
+    WHERE r.device = d.device
+    ORDER BY ts DESC LIMIT 1) x
+ORDER BY d.device;
+```
+
+**Do not use the second form without the index.** It then reads the whole table
+once for each device. On 10 million rows and 1,000 devices, the query read
+1,013 blocks in 45 ms with the index. Without it, the same query read 8.7 million
+blocks in 12.6 minutes.
+
+`DISTINCT ON (device) ... ORDER BY device, ts DESC` returns the same rows, but it
+sorts the whole table to do so. On the same 10 million rows it took 5.6 seconds.
+
+`test/latest_reading.sh` runs the SQL in this section and checks what it states.
+
 ## Add indexes and use index-only scans
 
 A columnar table supports B-tree and other index types. An index helps a highly

test/check_ledger.tsv

@@ -1652,6 +1652,14 @@ inheritance	inheritance	premise: while its leaf is an ordinary relation	15;16;17
 inheritance	inheritance	row identities match the heap	15;16;17;18;19	never	-
 inheritance	inheritance	the child is visible through the parent	15;16;17;18;19	never	-
 inheritance	inheritance	ungrouped vector aggregation also sees the child rows	15;16;17;18;19	never	-
+latest_reading	latest_reading	premise: the document's index exists	15;16;17;18;19	never	-
+latest_reading	latest_reading	premise: the document's three recipe blocks were found	15;16;17;18;19	never	-
+latest_reading	latest_reading	premise: the indexed LATERAL form read the table at all	15;16;17;18;19	never	-
+latest_reading	latest_reading	premise: the table holds every row	15;16;17;18;19	never	-
+latest_reading	latest_reading	the GROUP BY form returns what DISTINCT ON returns	15;16;17;18;19	never	-
+latest_reading	latest_reading	the LATERAL form returns what DISTINCT ON returns	15;16;17;18;19	2026-10-12	the document's LATERAL recipe orders by ts ascending, so it returns each device's FIRST reading
+latest_reading	latest_reading	with the btree the LATERAL form reads under a tenth of the table	15;16;17;18;19	never	-
+latest_reading	latest_reading	without the btree the LATERAL form reads the table once per device	15;16;17;18;19	never	-
 native_chunk_length_bound	native_chunk_length_bound	a sequential scan of the same poisoned chunk is refused (XX001)	15;16;17;18;19	never	-
 native_chunk_length_bound	native_chunk_length_bound	an index fetch of a chunk whose page_length is 2^32 too large is refused (XX001)	15;16;17;18;19	never	-
 native_chunk_length_bound	native_chunk_length_bound	backend survived the sequential refusal	15;16;17;18;19	never	-

test/check_ledger_budget.txt

@@ -1234,4 +1234,13 @@ suites_not_covered 247
 #     never         1651 + 2 = 1653
 #     dated          208 + 5 =  213
 #     1653 + 213 == 1866
-checks_never_observed_red 1653
+# #1293 (OffgridwithJD): 1653 -> 1660. latest_reading enters the ledger with 8
+# checks from a five-major run. One is dated: "the LATERAL form returns what
+# DISTINCT ON returns", by the document's recipe edited to order ascending, which
+# returns each device's first reading. The other 7 arrive `never`.
+#
+#     ledger rows   1866 + 8 = 1874
+#     never         1653 + 7 = 1660
+#     dated          213 + 1 =  214
+#     1660 + 214 == 1874
+checks_never_observed_red 1660

test/latest_reading.sh

@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+#
+# The "latest reading per device" recipe in docs/how-to.md does what it says (#1293).
+#
+# The recipe's SQL is READ FROM THE DOCUMENT, not copied here, so an edit to the
+# document is what this suite runs: a recipe that stops returning the right rows,
+# or stops being cheap, turns this red. The approach is jdatcmd's, from #1288.
+#
+# Measured on #1293 with 10,000,000 rows and 1,000 devices:
+#
+#     DISTINCT ON (device) ... ORDER BY device, ts DESC     17,204 blocks   5.6 s
+#     GROUP BY device, max(ts)                               8,071 blocks   1.1 s  (with a projection)
+#     a btree on (device, ts) + LATERAL over a device list   1,013 blocks   45 ms
+#     the same LATERAL WITHOUT that btree                8,733,021 blocks  757 s
+#
+# What is asserted, at a size a suite can afford. The LATERAL form costs about a
+# block per device, whatever the table's size, so the fixture needs many rows per
+# device for "a tenth of the table" to be the claim it is at scale: at 5,000 rows
+# per device it read 45 blocks of a 177-page table and failed; at 25,000 it is a
+# tenth with room.
+#
+#   1. each documented form returns exactly the rows DISTINCT ON returns;
+#   2. the btree + LATERAL form reads under a tenth of the table's blocks;
+#   3. the trap the document warns about holds: without the btree the LATERAL form
+#      reads the table once per device -- at least the table's blocks times half
+#      the device count.
+#
+# Usage:  test/latest_reading.sh [PG_CONFIG]
+
+set -uo pipefail
+. "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
+pgc_setup "${1:-/usr/local/pg17/bin/pg_config}"
+
+# at_least GOT FLOOR -- GOT when it falls short, else FLOOR; an empty GOT stays
+# empty, which check_num refuses (selftest part 540).
+at_least() { [ -z "$1" ] && return; [ "$1" -lt "$2" ] && echo "$1" || echo "$2"; }
+
+DOC="$PGC_SRCDIR/docs/how-to.md"
+# doc_sql N -> the Nth fenced sql block under "## Find the latest reading per device"
+doc_sql() {
+	awk -v want="$1" '
+		/^## /            { inside = ($0 == "## Find the latest reading per device") }
+		inside && /^```sql/ { n++; grab = (n == want); next }
+		/^```/            { grab = 0; next }
+		grab              { print }' "$DOC"
+}
+INDEX_SQL="$(doc_sql 1)"
+GROUP_SQL="$(doc_sql 2)"
+LATERAL_SQL="$(doc_sql 3)"
+found=0
+for b in "$INDEX_SQL" "$GROUP_SQL" "$LATERAL_SQL"; do [ -n "$b" ] && found=$((found + 1)); done
+check_num "premise: the document's three recipe blocks were found" "$found" "3"
+
+q "CREATE EXTENSION IF NOT EXISTS pgcolumnar;" >/dev/null
+DEVICES=40
+q "CREATE TABLE readings (device int, ts timestamptz, v float8) USING pgcolumnar;
+   CREATE TABLE devices (device int PRIMARY KEY);
+   INSERT INTO devices SELECT g FROM generate_series(1, $DEVICES) g;
+   INSERT INTO readings
+     SELECT 1 + (h % $DEVICES), '2026-05-01'::timestamptz + (h % 1000000) * interval '1 min', h % 977
+     FROM (SELECT hashint4(g)::bigint & 2147483647 AS h FROM generate_series(1, 1000000) g) x;" >/dev/null
+check_num "premise: the table holds every row" "$(q "SELECT count(*) FROM readings;")" "1000000"
+
+ORACLE="$(q "SELECT DISTINCT ON (device) device, ts FROM readings ORDER BY device, ts DESC;" | md5sum | cut -c1-12)"
+
+blocks() {	# blocks SQL -> readings blocks read by SQL, in its own session
+	q "SELECT pg_stat_reset();" >/dev/null
+	env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \
+		-d "$PGC_DB" -At -c "$1" -c "SELECT pg_stat_force_next_flush()" >/dev/null 2>&1
+	q "SELECT coalesce(heap_blks_read, 0) + coalesce(heap_blks_hit, 0)
+	   FROM pg_statio_user_tables WHERE relname = 'readings';"
+}
+pages="$(q "SELECT pg_relation_size('readings') / 8192;")"
+
+# The trap first, before the document's index exists.
+trap_blocks="$(blocks "$LATERAL_SQL")"
+echo "-- table $pages pages; LATERAL without the btree read $trap_blocks blocks"
+check_num "without the btree the LATERAL form reads the table once per device" \
+	"$( [ -n "$trap_blocks" ] && at_least "$trap_blocks" $((pages * DEVICES / 2)) )" "$((pages * DEVICES / 2))"
+
+check "the GROUP BY form returns what DISTINCT ON returns" \
+	"$(q "$GROUP_SQL" | md5sum | cut -c1-12)" "$ORACLE"
+
+q "$INDEX_SQL" >/dev/null
+check_num "premise: the document's index exists" \
+	"$(q "SELECT count(*) FROM pg_indexes WHERE tablename = 'readings';")" "1"
+check "the LATERAL form returns what DISTINCT ON returns" \
+	"$(q "$LATERAL_SQL" | md5sum | cut -c1-12)" "$ORACLE"
+lat_blocks="$(blocks "$LATERAL_SQL")"
+echo "-- with the btree the LATERAL form read $lat_blocks blocks"
+check_num "premise: the indexed LATERAL form read the table at all" \
+	"$(at_least "$lat_blocks" 1)" "1"
+# A query that reads NOTHING also reads under a tenth of the table: the red run
+# before the document existed passed this arm on empty SQL. So reading at least
+# one block is part of the claim, and a reading of zero is not a measurement.
+check_num "with the btree the LATERAL form reads under a tenth of the table" \
+	"$( [ -n "$lat_blocks" ] && [ "$lat_blocks" -ge 1 ] && at_least $((pages - 10 * lat_blocks)) 1 || echo "read ${lat_blocks:-nothing}" )" "1"
+
+pgc_summary

test/pytest/TESTS.md

@@ -132,6 +132,7 @@ behaviour, the source of that number is named.
 - [84. test_native_format.py: the format stamp is a range, and an append raises it](#84-test_native_formatpy-the-format-stamp-is-a-range-and-an-append-raises-it)
 - [85. test_native_agg.py: min/max over timestamptz read no data](#85-test_native_aggpy-minmax-over-timestamptz-read-no-data)
 - [86. test_format_downgrade.py: an alpha4 binary refuses what this build wrote](#86-test_format_downgradepy-an-alpha4-binary-refuses-what-this-build-wrote)
+- [87. test_latest_reading.py: the latest-reading recipe does what the document says](#87-test_latest_readingpy-the-latest-reading-recipe-does-what-the-document-says)
 
 ## 1. How to read a test in here
 
@@ -6383,3 +6384,23 @@ Removal proof: with `COLUMNAR_NATIVE_FORMAT_VERSION` set back to 1, the stamps r
 | --- | --- |
 | `test_an_alpha4_binary_refuses_what_this_build_wrote` | the old build stores both tables; the libraries differ; the upgrade succeeds and this build reads the old table; the stamps are `1 2 2` for read, appended and new; on the old library again, the table only read is still read and fetched by index, and the appended and new tables are refused with `0A000` on a scan and on an index fetch |
 | `test_the_library_swap_declines_beside_other_workers` | the decision that makes the swap decline beside other xdist workers, driven with the environments pytest-xdist sets: a serial run and a one-worker run proceed, a run beside three other workers declines, and a worker with no stated count is treated as not alone. A backend another worker started during the swap would load alpha4 against this tree's catalogs and fail in an unrelated test; atomic renames protect only processes that had already mapped the library (found by @jdatcmd). Under `-n`, the swap test declines as `UNMET_PRECONDITION` with that reason, so a parallel run ends INCOMPLETE naming it rather than risking someone else's test |
+
+## 87. test_latest_reading.py: the latest-reading recipe does what the document says
+
+Port of `latest_reading.sh` (#1293). The recipe's SQL is read out of `docs/how-to.md`'s "Find the latest reading per device" section, by this file's own extraction (Python, not the shell twin's awk). So an edit to the document is what runs: a recipe that stops returning the right rows, or stops being cheap, turns this red. The approach is jdatcmd's, from #1288.
+
+Measured on #1293 with 10 million rows and 1,000 devices: `DISTINCT ON` took 5.6 s, the btree plus `LATERAL` form 45 ms, and the same `LATERAL` without the btree 12.6 minutes. Asserted here: each documented form returns exactly the rows `DISTINCT ON` returns; with the btree the `LATERAL` form reads under a tenth of the table; and without it the table is read about once per device, which is the trap the document warns about.
+
+A ZERO IS NOT A MEASUREMENT. On the shell twin, the red run before the section existed passed "under a tenth of the table" on empty SQL: a query that reads nothing reads under a tenth of anything. Reading at least one block is now its own premise.
+
+THE FIXTURE NEEDS MANY ROWS PER DEVICE. The `LATERAL` form costs about a block per device whatever the table's size, so at 5,000 rows per device it read 45 blocks of a 177-page table and "a tenth" failed. Here it is 36,000 rows per device.
+
+On a private database, because the document names its tables, `readings` and `devices`.
+
+Removal proof: the document's `LATERAL` edited to `ORDER BY ts` (ascending) returns each device's first reading, and `the LATERAL form returns what DISTINCT ON returns` fails. On the shell twin that is the only red, on every major.
+
+### Every test
+
+| test | what it holds |
+| --- | --- |
+| `test_the_latest_reading_recipe_does_what_the_document_says` | the document's three blocks are found; the table holds its rows; without the btree the `LATERAL` form reads the table about once per device; the `GROUP BY` form and, with the document's index, the `LATERAL` form return the rows `DISTINCT ON` returns; the indexed `LATERAL` reads at least one block and under a tenth of the table |

test/pytest/expected_tests.txt

@@ -543,4 +543,4 @@ guard_tests 404
 # own test is not in it. Git conflicted again -- loudly, because the key moved
 # on both sides -- and the answer is neither branch's number. Collected on the
 # rebased tree, not added up.
-cluster_tests 494
+cluster_tests 495

test/pytest/test_compare_to_bash.py

@@ -102,6 +102,7 @@
             "index_am_support",
             "index_fetch_penalty_crossover",
             "index_fetch_penalty_width",
+            "latest_reading",
             "native_chunk_length_bound",
             "native_delete_vector_index",
             "native_delete_visibility_paths",

test/pytest/test_latest_reading.py

@@ -0,0 +1,110 @@
+"""The "latest reading per device" recipe in docs/how-to.md does what it says (#1293).
+
+The recipe's SQL is READ FROM THE DOCUMENT, so an edit to it is what runs: a
+recipe that stops returning the right rows, or stops being cheap, turns this red.
+Measured on #1293 with 10,000,000 rows and 1,000 devices: DISTINCT ON read 17,204
+blocks in 5.6 s; a btree on (device, ts) plus LATERAL read 1,013 blocks in 45 ms;
+the same LATERAL without the btree read 8.7 million blocks in 12.6 minutes.
+
+Asserted here: each documented form returns the rows DISTINCT ON returns; with
+the btree the LATERAL form reads under a tenth of the table, and at least one
+block (a query that reads nothing would otherwise pass); without it, it reads the
+table about once per device.
+
+ON A PRIVATE DATABASE (`pgc_own_db`): the document names its tables, `readings`
+and `devices`, so they cannot carry a per-test prefix. Independent of
+`test/latest_reading.sh`: its own extraction of the document (Python, not awk),
+its own data, device count and row count.
+"""
+
+import pathlib
+import re
+
+DOC = pathlib.Path(__file__).resolve().parent.parent.parent / "docs" / "how-to.md"
+SECTION = "## Find the latest reading per device"
+DEVICES = 25
+ROWS = 900000
+
+
+def _recipe_blocks():
+    text = DOC.read_text(encoding="utf-8")
+    start = text.find(SECTION + "\n")
+    if start < 0:
+        return []
+    rest = text[start + len(SECTION):]
+    nxt = re.search(r"^## ", rest, re.M)
+    body = rest[: nxt.start()] if nxt else rest
+    return [b.strip() for b in re.findall(r"^```sql\n(.*?)^```", body, re.M | re.S)]
+
+
+def _one(conn, sql):
+    with conn.cursor() as cur:
+        cur.execute(sql)
+        row = cur.fetchone()
+    return row[0] if row else None
+
+
+def _rows(conn, sql):
+    with conn.cursor() as cur:
+        cur.execute(sql)
+        return [tuple(map(str, r)) for r in cur.fetchall()]
+
+
+def _blocks(conn, sql):
+    with conn.cursor() as cur:
+        cur.execute("SELECT pg_stat_force_next_flush()")
+        cur.execute("SELECT pg_stat_reset()")
+        cur.execute(sql)
+        cur.fetchall()
+        cur.execute("SELECT pg_stat_force_next_flush()")
+        cur.execute("SELECT coalesce(heap_blks_read,0) + coalesce(heap_blks_hit,0)"
+                    " FROM pg_statio_user_tables WHERE relname = 'readings'")
+        row = cur.fetchone()
+    return int(row[0]) if row else None
+
+
+def test_the_latest_reading_recipe_does_what_the_document_says(pgc_own_db, expect):
+    conn = pgc_own_db
+    blocks = _recipe_blocks()
+    expect.num(len(blocks), 3, "premise: the document's three recipe blocks were found")
+    if len(blocks) != 3:
+        return
+    index_sql, group_sql, lateral_sql = blocks
+
+    with conn.cursor() as cur:
+        cur.execute("CREATE TABLE readings (device int, ts timestamptz, v float8) USING pgcolumnar")
+        cur.execute("CREATE TABLE devices (device int PRIMARY KEY)")
+        cur.execute(f"INSERT INTO devices SELECT g FROM generate_series(1, {DEVICES}) g")
+        cur.execute(
+            f"INSERT INTO readings SELECT 1 + (h % {DEVICES}), '2025-12-01'::timestamptz"
+            f" + (h % {ROWS}) * interval '30 s', (h % 613)::float8"
+            f" FROM (SELECT (hashint8(g) & 9223372036854775807) AS h"
+            f"       FROM generate_series(1::bigint, {ROWS}) g) x"
+        )
+    expect.num(_one(conn, "SELECT count(*) FROM readings"), ROWS,
+               "premise: the table holds every row")
+
+    oracle = _rows(conn, "SELECT DISTINCT ON (device) device, ts FROM readings"
+                         " ORDER BY device, ts DESC")
+    pages = _one(conn, "SELECT pg_relation_size('readings') / 8192")
+
+    trap = _blocks(conn, lateral_sql)
+    print(f"-- table {pages} pages; LATERAL without the btree read {trap} blocks")
+    expect.at_least(trap, pages * DEVICES // 2,
+                    "without the btree the LATERAL form reads the table once per device")
+
+    expect.text(str(_rows(conn, group_sql)), str(oracle),
+                "the GROUP BY form returns what DISTINCT ON returns")
+
+    with conn.cursor() as cur:
+        cur.execute(index_sql)
+    expect.num(_one(conn, "SELECT count(*) FROM pg_indexes WHERE tablename = 'readings'"), 1,
+               "premise: the document's index exists")
+    expect.text(str(_rows(conn, lateral_sql)), str(oracle),
+                "the LATERAL form returns what DISTINCT ON returns")
+    lat = _blocks(conn, lateral_sql)
+    print(f"-- with the btree the LATERAL form read {lat} blocks")
+    # Reading nothing also reads under a tenth of the table, so a zero is refused.
+    expect.at_least(lat, 1, "premise: the indexed LATERAL form read the table at all")
+    expect.at_least(pages - 10 * lat, 1,
+                    "with the btree the LATERAL form reads under a tenth of the table")

test/run_all_versions.sh

@@ -127,6 +127,7 @@ SUITES=(
 	inheritance
 	int8_agg_int128
 	isolation
+	latest_reading
 	local_open_race_free
 	logical_decoding_cdc_recipe
 	logical_decoding_source