From 39c53ffe182b872b0077ac5995fe3b1acfb9dca9 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 20:46:00 +0000 Subject: [PATCH] docs, test: the latest-reading-per-device recipe, run from the document (#1293) docs/how-to.md gains "Find the latest reading per device": GROUP BY for the latest timestamp, a btree on (device, ts) plus LATERAL over a device list for the latest row, and the trap -- the LATERAL form without the btree reads the whole table once per device. Measured on #1293, 10M rows and 1,000 devices: 45 ms with the btree, 12.6 minutes without, 5.6 s for DISTINCT ON. latest_reading.sh and its pytest twin read the recipe's SQL from the document, so an edit to the document is what runs (jdatcmd's approach from #1288): each documented form returns the rows DISTINCT ON returns with the btree, LATERAL reads >= 1 block and under a tenth of the table without it, LATERAL reads >= table pages x devices / 2 Red first, before the section existed: every arm red -- except "under a tenth of the table", which passed on empty SQL. A zero now is not a measurement. Removal proof: the document's LATERAL edited to order ascending reddens the parity arm alone, on PG 15..19, and in the pytest twin. latest_reading 8 passed, PG 15..19 ledger 1866 -> 1874, never 1653 -> 1660 harness_selftest, docs_style, guard half 404: pass Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 12 +++ docs/how-to.md | 42 +++++++++++ test/check_ledger.tsv | 8 ++ test/check_ledger_budget.txt | 11 ++- test/latest_reading.sh | 99 +++++++++++++++++++++++++ test/pytest/TESTS.md | 21 ++++++ test/pytest/expected_tests.txt | 2 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_latest_reading.py | 110 ++++++++++++++++++++++++++++ test/run_all_versions.sh | 1 + 10 files changed, 305 insertions(+), 2 deletions(-) create mode 100755 test/latest_reading.sh create mode 100644 test/pytest/test_latest_reading.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 073aab09a..62517d4d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,18 @@ true until the next version shipped. parity again after the newest and oldest rows are deleted. The first draft of the fixture overflowed `int`, its `INSERT` failed without a word, and every arm measured an empty table. A row-count premise now comes first. +- `docs/how-to.md` has a recipe for the latest reading per device (#1293), and + `test/latest_reading.sh` with its pytest twin runs the recipe's SQL straight from + the document. Each documented form must return the rows `DISTINCT ON` returns; + with the documented btree the `LATERAL` form must read under a tenth of the + table; and without it, the trap the section warns about must hold: the table is + read about once per device. Measured on #1293 with 10 million rows and 1,000 + devices: 45 ms with the btree, 12.6 minutes without it, and 5.6 seconds for + `DISTINCT ON`. Editing the document's recipe to order ascending reddens the + parity arm on every major. The first red run, before the section existed, also + showed a vacuous arm: "under a tenth of the table" passed on empty SQL, because + a query that reads nothing reads under a tenth of anything. It now requires at + least one block read. ### Changed diff --git a/docs/how-to.md b/docs/how-to.md index 03b17313f..ad47e5d67 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -207,6 +207,48 @@ serves poorly. Confirm the plan shows `Columnar Projection`. Keep the column lis minimal, because a projection that misses one queried column is never used. See Projections in the administration guide for detail. +## Find the latest reading per device + +"What is the latest reading for each device?" is the usual time-series question. +`max(ts)` over a whole table is answered from zone maps and reads almost no data. +Per device, choose one of two forms. + +To answer it fast for many devices, add a btree on the device and the timestamp: + +```sql +CREATE INDEX readings_device_ts ON readings (device, ts); +``` + +When you need only the latest timestamp, group by device. This reads the table +once: + +```sql +SELECT device, max(ts) FROM readings GROUP BY device ORDER BY device; +``` + +When you need the whole latest row, look each device up through the index. The +`devices` table lists the devices, one row each: + +```sql +SELECT d.device, x.ts +FROM devices d +CROSS JOIN LATERAL ( + SELECT ts FROM readings r + WHERE r.device = d.device + ORDER BY ts DESC LIMIT 1) x +ORDER BY d.device; +``` + +**Do not use the second form without the index.** It then reads the whole table +once for each device. On 10 million rows and 1,000 devices, the query read +1,013 blocks in 45 ms with the index. Without it, the same query read 8.7 million +blocks in 12.6 minutes. + +`DISTINCT ON (device) ... ORDER BY device, ts DESC` returns the same rows, but it +sorts the whole table to do so. On the same 10 million rows it took 5.6 seconds. + +`test/latest_reading.sh` runs the SQL in this section and checks what it states. + ## Add indexes and use index-only scans A columnar table supports B-tree and other index types. An index helps a highly diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index fb67dba10..905afced6 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1652,6 +1652,14 @@ inheritance inheritance premise: while its leaf is an ordinary relation 15;16;17 inheritance inheritance row identities match the heap 15;16;17;18;19 never - inheritance inheritance the child is visible through the parent 15;16;17;18;19 never - inheritance inheritance ungrouped vector aggregation also sees the child rows 15;16;17;18;19 never - +latest_reading latest_reading premise: the document's index exists 15;16;17;18;19 never - +latest_reading latest_reading premise: the document's three recipe blocks were found 15;16;17;18;19 never - +latest_reading latest_reading premise: the indexed LATERAL form read the table at all 15;16;17;18;19 never - +latest_reading latest_reading premise: the table holds every row 15;16;17;18;19 never - +latest_reading latest_reading the GROUP BY form returns what DISTINCT ON returns 15;16;17;18;19 never - +latest_reading latest_reading the LATERAL form returns what DISTINCT ON returns 15;16;17;18;19 2026-10-12 the document's LATERAL recipe orders by ts ascending, so it returns each device's FIRST reading +latest_reading latest_reading with the btree the LATERAL form reads under a tenth of the table 15;16;17;18;19 never - +latest_reading latest_reading without the btree the LATERAL form reads the table once per device 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound a sequential scan of the same poisoned chunk is refused (XX001) 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound an index fetch of a chunk whose page_length is 2^32 too large is refused (XX001) 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound backend survived the sequential refusal 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index fa032f292..0a7b69f2a 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -1234,4 +1234,13 @@ suites_not_covered 247 # never 1651 + 2 = 1653 # dated 208 + 5 = 213 # 1653 + 213 == 1866 -checks_never_observed_red 1653 +# #1293 (OffgridwithJD): 1653 -> 1660. latest_reading enters the ledger with 8 +# checks from a five-major run. One is dated: "the LATERAL form returns what +# DISTINCT ON returns", by the document's recipe edited to order ascending, which +# returns each device's first reading. The other 7 arrive `never`. +# +# ledger rows 1866 + 8 = 1874 +# never 1653 + 7 = 1660 +# dated 213 + 1 = 214 +# 1660 + 214 == 1874 +checks_never_observed_red 1660 diff --git a/test/latest_reading.sh b/test/latest_reading.sh new file mode 100755 index 000000000..035a0f241 --- /dev/null +++ b/test/latest_reading.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# +# The "latest reading per device" recipe in docs/how-to.md does what it says (#1293). +# +# The recipe's SQL is READ FROM THE DOCUMENT, not copied here, so an edit to the +# document is what this suite runs: a recipe that stops returning the right rows, +# or stops being cheap, turns this red. The approach is jdatcmd's, from #1288. +# +# Measured on #1293 with 10,000,000 rows and 1,000 devices: +# +# DISTINCT ON (device) ... ORDER BY device, ts DESC 17,204 blocks 5.6 s +# GROUP BY device, max(ts) 8,071 blocks 1.1 s (with a projection) +# a btree on (device, ts) + LATERAL over a device list 1,013 blocks 45 ms +# the same LATERAL WITHOUT that btree 8,733,021 blocks 757 s +# +# What is asserted, at a size a suite can afford. The LATERAL form costs about a +# block per device, whatever the table's size, so the fixture needs many rows per +# device for "a tenth of the table" to be the claim it is at scale: at 5,000 rows +# per device it read 45 blocks of a 177-page table and failed; at 25,000 it is a +# tenth with room. +# +# 1. each documented form returns exactly the rows DISTINCT ON returns; +# 2. the btree + LATERAL form reads under a tenth of the table's blocks; +# 3. the trap the document warns about holds: without the btree the LATERAL form +# reads the table once per device -- at least the table's blocks times half +# the device count. +# +# Usage: test/latest_reading.sh [PG_CONFIG] + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +# at_least GOT FLOOR -- GOT when it falls short, else FLOOR; an empty GOT stays +# empty, which check_num refuses (selftest part 540). +at_least() { [ -z "$1" ] && return; [ "$1" -lt "$2" ] && echo "$1" || echo "$2"; } + +DOC="$PGC_SRCDIR/docs/how-to.md" +# doc_sql N -> the Nth fenced sql block under "## Find the latest reading per device" +doc_sql() { + awk -v want="$1" ' + /^## / { inside = ($0 == "## Find the latest reading per device") } + inside && /^```sql/ { n++; grab = (n == want); next } + /^```/ { grab = 0; next } + grab { print }' "$DOC" +} +INDEX_SQL="$(doc_sql 1)" +GROUP_SQL="$(doc_sql 2)" +LATERAL_SQL="$(doc_sql 3)" +found=0 +for b in "$INDEX_SQL" "$GROUP_SQL" "$LATERAL_SQL"; do [ -n "$b" ] && found=$((found + 1)); done +check_num "premise: the document's three recipe blocks were found" "$found" "3" + +q "CREATE EXTENSION IF NOT EXISTS pgcolumnar;" >/dev/null +DEVICES=40 +q "CREATE TABLE readings (device int, ts timestamptz, v float8) USING pgcolumnar; + CREATE TABLE devices (device int PRIMARY KEY); + INSERT INTO devices SELECT g FROM generate_series(1, $DEVICES) g; + INSERT INTO readings + SELECT 1 + (h % $DEVICES), '2026-05-01'::timestamptz + (h % 1000000) * interval '1 min', h % 977 + FROM (SELECT hashint4(g)::bigint & 2147483647 AS h FROM generate_series(1, 1000000) g) x;" >/dev/null +check_num "premise: the table holds every row" "$(q "SELECT count(*) FROM readings;")" "1000000" + +ORACLE="$(q "SELECT DISTINCT ON (device) device, ts FROM readings ORDER BY device, ts DESC;" | md5sum | cut -c1-12)" + +blocks() { # blocks SQL -> readings blocks read by SQL, in its own session + q "SELECT pg_stat_reset();" >/dev/null + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -At -c "$1" -c "SELECT pg_stat_force_next_flush()" >/dev/null 2>&1 + q "SELECT coalesce(heap_blks_read, 0) + coalesce(heap_blks_hit, 0) + FROM pg_statio_user_tables WHERE relname = 'readings';" +} +pages="$(q "SELECT pg_relation_size('readings') / 8192;")" + +# The trap first, before the document's index exists. +trap_blocks="$(blocks "$LATERAL_SQL")" +echo "-- table $pages pages; LATERAL without the btree read $trap_blocks blocks" +check_num "without the btree the LATERAL form reads the table once per device" \ + "$( [ -n "$trap_blocks" ] && at_least "$trap_blocks" $((pages * DEVICES / 2)) )" "$((pages * DEVICES / 2))" + +check "the GROUP BY form returns what DISTINCT ON returns" \ + "$(q "$GROUP_SQL" | md5sum | cut -c1-12)" "$ORACLE" + +q "$INDEX_SQL" >/dev/null +check_num "premise: the document's index exists" \ + "$(q "SELECT count(*) FROM pg_indexes WHERE tablename = 'readings';")" "1" +check "the LATERAL form returns what DISTINCT ON returns" \ + "$(q "$LATERAL_SQL" | md5sum | cut -c1-12)" "$ORACLE" +lat_blocks="$(blocks "$LATERAL_SQL")" +echo "-- with the btree the LATERAL form read $lat_blocks blocks" +check_num "premise: the indexed LATERAL form read the table at all" \ + "$(at_least "$lat_blocks" 1)" "1" +# A query that reads NOTHING also reads under a tenth of the table: the red run +# before the document existed passed this arm on empty SQL. So reading at least +# one block is part of the claim, and a reading of zero is not a measurement. +check_num "with the btree the LATERAL form reads under a tenth of the table" \ + "$( [ -n "$lat_blocks" ] && [ "$lat_blocks" -ge 1 ] && at_least $((pages - 10 * lat_blocks)) 1 || echo "read ${lat_blocks:-nothing}" )" "1" + +pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 7f63e77f4..b3b165873 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -132,6 +132,7 @@ behaviour, the source of that number is named. - [84. test_native_format.py: the format stamp is a range, and an append raises it](#84-test_native_formatpy-the-format-stamp-is-a-range-and-an-append-raises-it) - [85. test_native_agg.py: min/max over timestamptz read no data](#85-test_native_aggpy-minmax-over-timestamptz-read-no-data) - [86. test_format_downgrade.py: an alpha4 binary refuses what this build wrote](#86-test_format_downgradepy-an-alpha4-binary-refuses-what-this-build-wrote) +- [87. test_latest_reading.py: the latest-reading recipe does what the document says](#87-test_latest_readingpy-the-latest-reading-recipe-does-what-the-document-says) ## 1. How to read a test in here @@ -6383,3 +6384,23 @@ Removal proof: with `COLUMNAR_NATIVE_FORMAT_VERSION` set back to 1, the stamps r | --- | --- | | `test_an_alpha4_binary_refuses_what_this_build_wrote` | the old build stores both tables; the libraries differ; the upgrade succeeds and this build reads the old table; the stamps are `1 2 2` for read, appended and new; on the old library again, the table only read is still read and fetched by index, and the appended and new tables are refused with `0A000` on a scan and on an index fetch | | `test_the_library_swap_declines_beside_other_workers` | the decision that makes the swap decline beside other xdist workers, driven with the environments pytest-xdist sets: a serial run and a one-worker run proceed, a run beside three other workers declines, and a worker with no stated count is treated as not alone. A backend another worker started during the swap would load alpha4 against this tree's catalogs and fail in an unrelated test; atomic renames protect only processes that had already mapped the library (found by @jdatcmd). Under `-n`, the swap test declines as `UNMET_PRECONDITION` with that reason, so a parallel run ends INCOMPLETE naming it rather than risking someone else's test | + +## 87. test_latest_reading.py: the latest-reading recipe does what the document says + +Port of `latest_reading.sh` (#1293). The recipe's SQL is read out of `docs/how-to.md`'s "Find the latest reading per device" section, by this file's own extraction (Python, not the shell twin's awk). So an edit to the document is what runs: a recipe that stops returning the right rows, or stops being cheap, turns this red. The approach is jdatcmd's, from #1288. + +Measured on #1293 with 10 million rows and 1,000 devices: `DISTINCT ON` took 5.6 s, the btree plus `LATERAL` form 45 ms, and the same `LATERAL` without the btree 12.6 minutes. Asserted here: each documented form returns exactly the rows `DISTINCT ON` returns; with the btree the `LATERAL` form reads under a tenth of the table; and without it the table is read about once per device, which is the trap the document warns about. + +A ZERO IS NOT A MEASUREMENT. On the shell twin, the red run before the section existed passed "under a tenth of the table" on empty SQL: a query that reads nothing reads under a tenth of anything. Reading at least one block is now its own premise. + +THE FIXTURE NEEDS MANY ROWS PER DEVICE. The `LATERAL` form costs about a block per device whatever the table's size, so at 5,000 rows per device it read 45 blocks of a 177-page table and "a tenth" failed. Here it is 36,000 rows per device. + +On a private database, because the document names its tables, `readings` and `devices`. + +Removal proof: the document's `LATERAL` edited to `ORDER BY ts` (ascending) returns each device's first reading, and `the LATERAL form returns what DISTINCT ON returns` fails. On the shell twin that is the only red, on every major. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_latest_reading_recipe_does_what_the_document_says` | the document's three blocks are found; the table holds its rows; without the btree the `LATERAL` form reads the table about once per device; the `GROUP BY` form and, with the document's index, the `LATERAL` form return the rows `DISTINCT ON` returns; the indexed `LATERAL` reads at least one block and under a tenth of the table | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 3d4aa087b..ade202328 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -543,4 +543,4 @@ guard_tests 404 # own test is not in it. Git conflicted again -- loudly, because the key moved # on both sides -- and the answer is neither branch's number. Collected on the # rebased tree, not added up. -cluster_tests 494 +cluster_tests 495 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index a3fe4bc65..63206b037 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -102,6 +102,7 @@ "index_am_support", "index_fetch_penalty_crossover", "index_fetch_penalty_width", + "latest_reading", "native_chunk_length_bound", "native_delete_vector_index", "native_delete_visibility_paths", diff --git a/test/pytest/test_latest_reading.py b/test/pytest/test_latest_reading.py new file mode 100644 index 000000000..582d529bf --- /dev/null +++ b/test/pytest/test_latest_reading.py @@ -0,0 +1,110 @@ +"""The "latest reading per device" recipe in docs/how-to.md does what it says (#1293). + +The recipe's SQL is READ FROM THE DOCUMENT, so an edit to it is what runs: a +recipe that stops returning the right rows, or stops being cheap, turns this red. +Measured on #1293 with 10,000,000 rows and 1,000 devices: DISTINCT ON read 17,204 +blocks in 5.6 s; a btree on (device, ts) plus LATERAL read 1,013 blocks in 45 ms; +the same LATERAL without the btree read 8.7 million blocks in 12.6 minutes. + +Asserted here: each documented form returns the rows DISTINCT ON returns; with +the btree the LATERAL form reads under a tenth of the table, and at least one +block (a query that reads nothing would otherwise pass); without it, it reads the +table about once per device. + +ON A PRIVATE DATABASE (`pgc_own_db`): the document names its tables, `readings` +and `devices`, so they cannot carry a per-test prefix. Independent of +`test/latest_reading.sh`: its own extraction of the document (Python, not awk), +its own data, device count and row count. +""" + +import pathlib +import re + +DOC = pathlib.Path(__file__).resolve().parent.parent.parent / "docs" / "how-to.md" +SECTION = "## Find the latest reading per device" +DEVICES = 25 +ROWS = 900000 + + +def _recipe_blocks(): + text = DOC.read_text(encoding="utf-8") + start = text.find(SECTION + "\n") + if start < 0: + return [] + rest = text[start + len(SECTION):] + nxt = re.search(r"^## ", rest, re.M) + body = rest[: nxt.start()] if nxt else rest + return [b.strip() for b in re.findall(r"^```sql\n(.*?)^```", body, re.M | re.S)] + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + row = cur.fetchone() + return row[0] if row else None + + +def _rows(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return [tuple(map(str, r)) for r in cur.fetchall()] + + +def _blocks(conn, sql): + with conn.cursor() as cur: + cur.execute("SELECT pg_stat_force_next_flush()") + cur.execute("SELECT pg_stat_reset()") + cur.execute(sql) + cur.fetchall() + cur.execute("SELECT pg_stat_force_next_flush()") + cur.execute("SELECT coalesce(heap_blks_read,0) + coalesce(heap_blks_hit,0)" + " FROM pg_statio_user_tables WHERE relname = 'readings'") + row = cur.fetchone() + return int(row[0]) if row else None + + +def test_the_latest_reading_recipe_does_what_the_document_says(pgc_own_db, expect): + conn = pgc_own_db + blocks = _recipe_blocks() + expect.num(len(blocks), 3, "premise: the document's three recipe blocks were found") + if len(blocks) != 3: + return + index_sql, group_sql, lateral_sql = blocks + + with conn.cursor() as cur: + cur.execute("CREATE TABLE readings (device int, ts timestamptz, v float8) USING pgcolumnar") + cur.execute("CREATE TABLE devices (device int PRIMARY KEY)") + cur.execute(f"INSERT INTO devices SELECT g FROM generate_series(1, {DEVICES}) g") + cur.execute( + f"INSERT INTO readings SELECT 1 + (h % {DEVICES}), '2025-12-01'::timestamptz" + f" + (h % {ROWS}) * interval '30 s', (h % 613)::float8" + f" FROM (SELECT (hashint8(g) & 9223372036854775807) AS h" + f" FROM generate_series(1::bigint, {ROWS}) g) x" + ) + expect.num(_one(conn, "SELECT count(*) FROM readings"), ROWS, + "premise: the table holds every row") + + oracle = _rows(conn, "SELECT DISTINCT ON (device) device, ts FROM readings" + " ORDER BY device, ts DESC") + pages = _one(conn, "SELECT pg_relation_size('readings') / 8192") + + trap = _blocks(conn, lateral_sql) + print(f"-- table {pages} pages; LATERAL without the btree read {trap} blocks") + expect.at_least(trap, pages * DEVICES // 2, + "without the btree the LATERAL form reads the table once per device") + + expect.text(str(_rows(conn, group_sql)), str(oracle), + "the GROUP BY form returns what DISTINCT ON returns") + + with conn.cursor() as cur: + cur.execute(index_sql) + expect.num(_one(conn, "SELECT count(*) FROM pg_indexes WHERE tablename = 'readings'"), 1, + "premise: the document's index exists") + expect.text(str(_rows(conn, lateral_sql)), str(oracle), + "the LATERAL form returns what DISTINCT ON returns") + lat = _blocks(conn, lateral_sql) + print(f"-- with the btree the LATERAL form read {lat} blocks") + # Reading nothing also reads under a tenth of the table, so a zero is refused. + expect.at_least(lat, 1, "premise: the indexed LATERAL form read the table at all") + expect.at_least(pages - 10 * lat, 1, + "with the btree the LATERAL form reads under a tenth of the table") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index f592d4931..5c01d7657 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -127,6 +127,7 @@ SUITES=( inheritance int8_agg_int128 isolation + latest_reading local_open_race_free logical_decoding_cdc_recipe logical_decoding_source