[OffgridwithJD]

A ctid range reads only the row groups that hold it

Follows from the #1294 design note. Before this, the access method had no scan_set_tidrange, so a ctid range was a per-row filter over the whole table. On #1294, WHERE ctid >= '(600000,0)' on a 10M-row table matched 0 rows and still read all 67 row groups. The rollup design there needs "the newest row groups only".

What changed (src/columnar_tableam.c, no SQL change):

• scan_set_tidrange maps the bounds to row numbers and restricts the reader to the row groups that intersect them, through the existing PgColumnarReadRestrictToGroups.
• scan_getnextslot_tidrange filters the rows of the boundary groups. It does not stop at the first row past the upper bound, because compaction moves rows into later groups.
• A rescan ends the old reader, so a new range builds a fresh one.
• PostgreSQL 19's parallel TID range scan shares the reader's group claim. That needed no code of its own; an arm proves it.

Measured: on 400,000 rows, the last 2.5% by ctid reads 28 blocks, against 923 for the whole table. The instrument is pg_statio under a forced decode (max(v || 'x')), because an unfiltered max is answered from zone maps.

The bounds no row has. A block holds 291 rows, at offsets 1 to 291, but a bound can name offset 0 or anything up to 65535. The executor turns ctid < '(B,0)' into an upper bound of (B-1,65535). The first build read those offsets unclamped and returned rows from the next block. Arms written for those bounds went red, which is how it was found. The fix clamps them.

Tests (written first; red on main at the cost arm)

• test/tid_range_scan.sh: 29/29 on PG15–18 plus one named SKIP, 31/31 on PG19. The SKIP is the parallel arm, since PG15–18 have no parallel TID range scan.
• test/pytest/test_tid_range_scan.py, independent of the shell suite: 30/30 on PG19, 26/26 + 1 UNRUN on PG17. The parallel test is UNSUPPORTED_MAJOR below 19 and listed in expected_unrunnable.txt. compare_to_bash: 13 literal and 2 template matches, 0 missing.
• The oracle is the same query with enable_tidscan = off. Every plan premise is taken on the very query compared. A draft whose lateral output went unreferenced planned no scan at all, and its premise caught that.

Removal proofs, each red on its named arm in both twins: no group restriction (the cost arm); an unclamped upper offset past 291, an unclamped lower offset past 291, an unclamped lower offset 0, and a block-0 upper bound read as the last row (the parity arms for those bounds); no boundary-row filter; a reader kept across a rescan (the rescan arm); and no shared parallel claim on PG19 (it returned every row twice, 480002 vs 240001).

One unexplained result, recorded rather than dropped: an early mutant of the offset-0 upper bound did not go red. A later mutant of the same branch (C4) does, so the arm reaches that code, but I never explained why the first one survived.

Known limit, a follow-up: the custom scan still treats ctid as a per-row filter. When parallelism is cheap, the planner prefers a parallel custom scan and reads everything. At default settings the TID range scan is chosen, and a premise checks that.

Registries

• Ledger: 31 rows from a five-major run, replayed onto fix: a missing pyarrow declines a suite instead of failing it (#1215) #1286's ledger. 15 are dated by those mutations, each merged one log at a time with --mutation and --target. Census 1674 → 1690. The 16 never rows are 9 premises and 7 parity arms that no mutant of this change reddens.
• TESTS.md section 92; cluster_tests 499 and guard_tests 413, both by collection. The guard half passes 413.
• The gate reports suites_not_covered 249 > 247 with my suite-list extraction, and gives the same 249 on main. This change adds one registered and one covered suite, so it doesn't move that number. I'm flagging it in case my extraction differs from run_all_versions.sh's.

🤖 Generated with Claude Code

[jdatcmd]

Review of e568782. The range arithmetic, the restriction and the parallel path hold up. One regression is blocking: a scroll cursor over a ctid range now errors on a backward fetch, where on main it works.

Blocking: a backward fetch through a TID range scan errors

Measured on PG16, built from this PR's head, with a 5000-row table in 1000-row groups:

BEGIN;
DECLARE c SCROLL CURSOR FOR SELECT id FROM st WHERE ctid >= '(0,1)';
FETCH LAST FROM c;
-- ERROR:  columnar tables do not support backward scans
-- DETAIL: A scrollable cursor over relation "st" asked for a backward fetch, which native storage cannot serve.

The same cursor with SET enable_tidscan = off, which gives main's plan shape (main has no TID range callbacks, so it can't plan one), returns 5500, 5499, 5498 for FETCH LAST then FETCH BACKWARD 2.

Why it happens: ExecSupportsBackwardScan returns true for T_TidRangeScan (execAmi.c, beside T_SeqScan). So the planner puts no Material under a scroll cursor, and the backward fetch reaches pgcolumnar_scan_getnextslot. A sequential scan avoids this only because our custom scan replaces it, which is what the error's HINT relies on. A TID range scan isn't replaced, so this PR opens a path the HINT doesn't cover. The HINT's advice, "leave enable_custom_scan on", is already followed here and doesn't help, so the user has no way out but an ORDER BY.

Possible fixes, your call:

1. In the planner hook, when cursorOptions & CURSOR_OPT_SCROLL, drop TID range paths on columnar rels, or make sure a Material goes above them. This is the narrowest fix.
2. Serve backward in getnextslot_tidrange by buffering the in-range rows. The range can be large, so I'd not.

Either way, add an arm that is red on e568782: a scroll cursor over a ctid range, FETCH LAST and FETCH BACKWARD, compared with the same cursor under enable_tidscan = off. Both twins.

Verified, no change needed

• Rows inserted earlier in the same transaction are seen. In BEGIN; INSERT 500 more; SELECT count(*) ... WHERE ctid >= '(0,1)' the plan is Tid Range Scan and the count is 5500, equal to the filter's 5500. If that's not already an arm, it's worth one: set_tidrange reads the row-group catalog, and the arm proves the pending writes are in it by then.
• PG19 parallel composes correctly. PG19 plans partial TID range paths for any AM with the callbacks (tidpath.c, AMFLAG_HAS_TID_RANGE). Each worker calls set_tidrange under the same restored snapshot, so every worker builds the same rangeGroups. The claim loop (reader.c:2784) skips a claimed group outside the restriction, so each group is claimed by exactly one worker and read only if it's in range. Your parallel arm, with its PG15–18 skip, covers it.
• The offset edge cases (offset 0, past COLUMNAR_VALID_ITEMPOINTER_OFFSETS, and below row 0 at block 0) match the comment, and not stopping at the first out-of-range row is right after compaction moves rows.

I'll re-review once the cursor arm is in and shown red.

[OffgridwithJD]

Thanks, that was a real regression. Fixed at 23a25ab, rebased onto dd08f4e (#1304).

The fix (fb2f2aa) is your option 1, done after planning. A planner_hook checks a cursor that may scroll, which means CURSOR_OPT_SCROLL, or a DECLARE (CURSOR_OPT_FAST_PLAN) that didn't say NO SCROLL. If its plan holds a TID range scan of a columnar table, the hook puts a Material on top, the same thing standard_planner does for a SCROLL cursor over a plan that can't run backward. The Material's target list is built the way set_plan_references would build it (OUTER_VAR per column), and it gets its own node id. No path is removed, so a plain query still plans the TID range scan.

The plain-DECLARE case was broken too. PerformCursorOpen makes a cursor without SCROLL or NO SCROLL scrollable when ExecSupportsBackwardScan(planTree) is true, which it is for a TID range scan, so a backward fetch on a plain cursor also reached the AM. With the filter's plan, that same cursor gets 55000 ("can only scan forward"). With the Material, it answers correctly. So the arm compares the plain cursor with a SCROLL cursor under enable_tidscan = off, not with the filter's plain cursor.

Arms, in both twins, red on e568782 for the right reason:

• a SCROLL cursor over a ctid range fetches backward as the filter's plan does: on e568782 it gave 0A000 at FETCH LAST; the filter's plan gave 170000 169999 169998 150003.
• a plain cursor over a ctid range answers a backward fetch correctly: also 0A000 on e568782.
• rows inserted earlier in the same transaction are in the range, as the filter sees them: your verified case, now an arm, with a plan premise taken inside the transaction.

Each plan premise is taken on the very cursor query that's compared.

Removal proofs: without the hook, both cursor arms are red; with the hook covering SCROLL only, the plain-cursor arm is red.

One mistake of mine, caught by the evidence run: PostgreSQL 19's planner_hook and standard_planner take an ExplainState * as well. I first believed the hook built on 19, but that build reused 17's objects, because I didn't make clean between majors. The five-major run failed to build on 19, which is how I found it. It's fixed with a version macro, and every major now builds after a clean.

Evidence on 23a25ab:

• tid_range_scan.sh: 34/34 plus the parallel SKIP on PG15–18, and 36/36 on PG19.
• The pytest twin: 31 pass + 1 UNRUN on PG15–18, 35 pass on PG19, and pytest exited 0 each time.
• harness_selftest PG17: 1223 passed. Guard half: 413 passed.
• Ledger replayed onto main's: 36 rows, 17 dated by ten mutations, census 1674 → 1693. The commit before it (cadd1c5) carries main's ledger unchanged; its message still cites the pre-rebase census, and 23a25ab holds the real one.

🤖 Generated with Claude Code

[jdatcmd]

Approved at 23a25ab. The planner-hook fix is the right shape, and catching the plain-DECLARE case was a good find: PerformCursorOpen makes a cursor scrollable when ExecSupportsBackwardScan says so, which my review missed.

Checked

• materialize_finished_plan is exported on every major: present in optimizer/planmain.h for 15, 16, 17, 18 and 19 in our installs.
• Hook chaining: pgcolumnar_prev_planner_hook is saved and called before ours, and the PG19 ExplainState parameter is behind the version macro.
• Cursor options: SCROLL, or FAST_PLAN without NO_SCROLL. Anything else returns unchanged, so a plain query still plans the TID range scan.
• The walker's coverage: lefttree, righttree, Append, MergeAppend, SubqueryScan and CustomScan's custom_plans. A TID range scan under a CteScan or a SubPlan is read forward by its tuplestore or by re-execution, so not walking stmt->subplans for the search is right.

Not blocking, please fix in a follow-up commit or here: the Material's plan_node_id can collide

mat->plan_node_id = maxid + 1, where maxid is the largest id in the main tree only. In PG19's planner.c, set_plan_references numbers the top plan first (line 643) and the subplans after it (line 651), from the same lastPlanNodeId counter (setrefs.c:650). So whenever the cursor's query has a SubPlan or InitPlan, its first node already holds maxid + 1, and the Material duplicates it.

plannodes.h documents plan_node_id as unique across the final plan tree. I found no core consumer that this breaks today: the Material sits above any Gather, so it never enters a worker's DSM toc or instrumentation array. That's why this isn't blocking. Extensions that key on node ids would see the duplicate, though.

The fix is to also run the max over stmt->subplans, about three lines. Example query: DECLARE c SCROLL CURSOR FOR SELECT id, (SELECT max(id) FROM st) FROM st WHERE ctid >= '(0,1)'. SQL can't observe the id, so I wouldn't ask for an arm. A one-off assert-build check that the ids in the final tree are unique, red before the fix and green after, would be the proof.

Evidence

I didn't re-run your suites: this box's five majors are busy with #1290's mutation runs. I'm relying on your posted runs (34/34 + SKIP on PG15–18, 36/36 on PG19, both twins) and on CI, which has 13 passed and 2 pending at the time of this review. I'll merge only when all 15 are green.

[jdatcmd]

Merging now: approved at 23a25ab, 15/15 green. The plan_node_id note in my review is a follow-up, not a blocker.