Skip to content

fix: twelve metadata lookups read their whole catalog (#1207) #1285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jdatcmd merged 1 commit into from
Oct 12, 2026
Merged

fix: twelve metadata lookups read their whole catalog (#1207) #1285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,49 @@ true until the next version shipped.

### Fixed

- Twelve metadata lookups no longer read their whole catalog to answer a
question about one table (#1207). Each passed `InvalidOid` to
`systable_beginscan` on a key an existing index starts with, and the
`pgcolumnar` catalogs are shared by every columnar table in the database, so
each paid for every other table's rows:

free_space the compaction reuse, both end-truncation reads
row_group row_group_exists, reached by EVERY DELETE
storage three existence checks on every write, and the
sorted-extent update
projection_declaration list, rename a column, drop one, drop all

Each now goes through `pgcolumnar_scan_index_oid()`, so it probes the index
once the catalog reaches `pgcolumnar.index_min_blocks` pages and reads the heap
below that, as the earlier conversions do (#1213, #1217). Measured on PG 17
with 400 other projected tables in the database, catalog buffers per
operation, before and after:

one-row DELETE row_group 45 -> 19
INSERT storage 19 -> 13
vacuum_sorted storage 128 -> 93
compact_rewrite free_space 43 -> 37
truncate() free_space 26 -> 17
RENAME COLUMN projection_declaration 9 -> 4

`test/catalog_scan_index.sh` and its pytest twin drive eleven operations
that between them reach all twelve sites. Each operation runs after the
population has put its catalog above the threshold, and each arm asserts the
operation made no sequential scan of that catalog. That is a path assertion,
which `catalog_delete_index.sh` warns against, and it is used here only where
a premise shows that declining the probe cannot be correct. Two cost-based
versions were tried first and a revert passed both. The full revert reddens
all eleven arms on PG 15 through 19. Reverting each site on its own reddens
the arm for the operation that reaches it.

Two side effects, neither a behaviour anyone relied on.
`PgColumnarAllocateFreeSpace` picks the smallest range that fits, and a tie
now goes to the lowest file offset instead of heap order. Projection
declarations are listed in name order instead of heap order.

The `server_dir` pytest fixture moved to `conftest.py`, because a second file
now needs a directory the server can write.

- `CREATE MATERIALIZED VIEW ... USING pgcolumnar ... WITH DATA` no longer
orphans `pgcolumnar.storage.relation_oid` (#1275). The statement builds a
transient relation, fills it and swaps, exactly as a rewriting `ALTER` does,
Expand Down
46 changes: 34 additions & 12 deletions src/columnar_metadata.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1028,6 +1028,7 @@ bool
PgColumnarAllocateFreeSpace(uint64 storageId, uint64 dataLength,
TransactionId oldestXmin, uint64 *fileOffset)
{
Oid scanIdx;
Relation rel = open_columnar_table("free_space", RowExclusiveLock);
TupleDesc td = RelationGetDescr(rel);
ScanKeyData key[1];
Expand All @@ -1044,7 +1045,8 @@ PgColumnarAllocateFreeSpace(uint64 storageId, uint64 dataLength,
snap = RegisterSnapshot(GetLatestSnapshot());
ScanKeyInit(&key[0], Anum_free_space_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) storageId));
scan = systable_beginscan(rel, InvalidOid, false, snap, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "free_space_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snap, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
bool isnull;
Expand Down Expand Up @@ -1245,6 +1247,7 @@ bool
PgColumnarTrailingFreeSpaceSafe(uint64 storageId, uint64 liveEnd,
TransactionId oldestXmin)
{
Oid scanIdx;
Relation rel = open_columnar_table("free_space", AccessShareLock);
TupleDesc td = RelationGetDescr(rel);
Snapshot snap = RegisterSnapshot(GetLatestSnapshot());
Expand All @@ -1255,7 +1258,8 @@ PgColumnarTrailingFreeSpaceSafe(uint64 storageId, uint64 liveEnd,

ScanKeyInit(&key[0], Anum_free_space_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) storageId));
scan = systable_beginscan(rel, InvalidOid, false, snap, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "free_space_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snap, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
bool isnull;
Expand Down Expand Up @@ -1289,6 +1293,7 @@ PgColumnarTrailingFreeSpaceSafe(uint64 storageId, uint64 liveEnd,
void
PgColumnarDeleteFreeSpaceAtOrAbove(uint64 storageId, uint64 liveEnd)
{
Oid scanIdx;
Relation rel = open_columnar_table("free_space", RowExclusiveLock);
TupleDesc td = RelationGetDescr(rel);
Snapshot snap = RegisterSnapshot(GetLatestSnapshot());
Expand All @@ -1300,7 +1305,8 @@ PgColumnarDeleteFreeSpaceAtOrAbove(uint64 storageId, uint64 liveEnd)

ScanKeyInit(&key[0], Anum_free_space_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) storageId));
scan = systable_beginscan(rel, InvalidOid, false, snap, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "free_space_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snap, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
bool isnull;
Expand Down Expand Up @@ -1748,6 +1754,7 @@ delete_vector_lock_chunk_group(uint64 storageId, uint64 stripeId, int chunkId)
static bool
row_group_exists(uint64 storageId, uint64 groupNumber, Snapshot snapshot)
{
Oid scanIdx;
Relation rel = open_columnar_table("row_group", AccessShareLock);
ScanKeyData key[2];
SysScanDesc scan;
Expand All @@ -1757,7 +1764,8 @@ row_group_exists(uint64 storageId, uint64 groupNumber, Snapshot snapshot)
F_INT8EQ, Int64GetDatum((int64) storageId));
ScanKeyInit(&key[1], Anum_row_group_group_number, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) groupNumber));
scan = systable_beginscan(rel, InvalidOid, false, snapshot, 2, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "row_group_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snapshot, 2, key);
found = HeapTupleIsValid(systable_getnext(scan));
systable_endscan(scan);
table_close(rel, AccessShareLock);
Expand Down Expand Up @@ -2031,6 +2039,7 @@ columnar_relation_is_new_in_xact(Oid relid)
void
PgColumnarInsertNativeStorageRow(const NativeStorageMetadata *s)
{
Oid scanIdx;
Relation rel = open_columnar_table("storage", RowExclusiveLock);
TupleDesc tupdesc = RelationGetDescr(rel);
Datum values[Natts_native_storage];
Expand Down Expand Up @@ -2074,7 +2083,8 @@ PgColumnarInsertNativeStorageRow(const NativeStorageMetadata *s)
snapshot = PgColumnarCatalogSnapshot(GetActiveSnapshot());
ScanKeyInit(&key[0], Anum_native_storage_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) s->storageId));
scan = systable_beginscan(rel, InvalidOid, false, snapshot, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "storage_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snapshot, 1, key);
exists = HeapTupleIsValid(systable_getnext(scan));
systable_endscan(scan);
PopActiveSnapshot();
Expand Down Expand Up @@ -2112,7 +2122,8 @@ PgColumnarInsertNativeStorageRow(const NativeStorageMetadata *s)
snapshot = PgColumnarCatalogSnapshot(GetActiveSnapshot());
ScanKeyInit(&key[0], Anum_native_storage_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) s->storageId));
scan = systable_beginscan(rel, InvalidOid, false, snapshot, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "storage_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snapshot, 1, key);
exists = HeapTupleIsValid(systable_getnext(scan));
systable_endscan(scan);
if (exists)
Expand Down Expand Up @@ -2144,7 +2155,8 @@ PgColumnarInsertNativeStorageRow(const NativeStorageMetadata *s)
snapshot = PgColumnarCatalogSnapshot(GetActiveSnapshot());
ScanKeyInit(&key[0], Anum_native_storage_storage_id, BTEqualStrategyNumber,
F_INT8EQ, Int64GetDatum((int64) s->storageId));
scan = systable_beginscan(rel, InvalidOid, false, snapshot, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "storage_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), snapshot, 1, key);
exists = HeapTupleIsValid(systable_getnext(scan));
systable_endscan(scan);
PopActiveSnapshot();
Expand Down Expand Up @@ -2474,6 +2486,7 @@ void
PgColumnarSetSortedExtent(uint64 storageId, int64 firstGroup, int64 lastGroup,
List *sortByNames, const char *sortedKind)
{
Oid scanIdx;
Relation rel;
TupleDesc tupdesc;
ScanKeyData key[1];
Expand Down Expand Up @@ -2513,7 +2526,8 @@ PgColumnarSetSortedExtent(uint64 storageId, int64 firstGroup, int64 lastGroup,
F_INT8EQ, Int64GetDatum((int64) storageId));

/* NULL snapshot -> catalog snapshot, as the other read-side scans use. */
scan = systable_beginscan(rel, InvalidOid, false, NULL, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "storage_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), NULL, 1, key);
tuple = systable_getnext(scan);
if (HeapTupleIsValid(tuple))
{
Expand Down Expand Up @@ -4071,6 +4085,7 @@ PgColumnarInsertProjectionRow(const PgColumnarProjection *proj)
List *
PgColumnarListProjectionDeclarations(Oid relid)
{
Oid scanIdx;
Relation rel = open_columnar_table("projection_declaration",
AccessShareLock);
TupleDesc tupdesc = RelationGetDescr(rel);
Expand All @@ -4082,7 +4097,8 @@ PgColumnarListProjectionDeclarations(Oid relid)
ScanKeyInit(&key[0], Anum_projection_declaration_rel, BTEqualStrategyNumber,
F_OIDEQ, ObjectIdGetDatum(relid));

scan = systable_beginscan(rel, InvalidOid, false, NULL, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "projection_declaration_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), NULL, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
PgColumnarProjectionDeclaration *d =
Expand Down Expand Up @@ -4213,6 +4229,7 @@ void
PgColumnarRenameProjectionDeclarationColumn(Oid relid, const char *oldName,
const char *newName)
{
Oid scanIdx;
Relation rel;
TupleDesc tupdesc;
ScanKeyData key[1];
Expand All @@ -4223,7 +4240,8 @@ PgColumnarRenameProjectionDeclarationColumn(Oid relid, const char *oldName,
tupdesc = RelationGetDescr(rel);
ScanKeyInit(&key[0], Anum_projection_declaration_rel, BTEqualStrategyNumber,
F_OIDEQ, ObjectIdGetDatum(relid));
scan = systable_beginscan(rel, InvalidOid, false, NULL, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "projection_declaration_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), NULL, 1, key);

while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
Expand Down Expand Up @@ -4288,6 +4306,7 @@ PgColumnarRenameProjectionDeclarationColumn(Oid relid, const char *oldName,
void
PgColumnarDeleteProjectionDeclaration(Oid relid, const char *name)
{
Oid scanIdx;
Relation rel = open_columnar_table("projection_declaration",
RowExclusiveLock);
ScanKeyData key[1];
Expand All @@ -4298,7 +4317,8 @@ PgColumnarDeleteProjectionDeclaration(Oid relid, const char *name)
ScanKeyInit(&key[0], Anum_projection_declaration_rel, BTEqualStrategyNumber,
F_OIDEQ, ObjectIdGetDatum(relid));

scan = systable_beginscan(rel, InvalidOid, false, NULL, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "projection_declaration_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), NULL, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
{
bool isnull;
Expand Down Expand Up @@ -4329,6 +4349,7 @@ PgColumnarDeleteProjectionDeclaration(Oid relid, const char *name)
void
PgColumnarDeleteProjectionDeclarationsForRel(Oid relid)
{
Oid scanIdx;
Relation rel = open_columnar_table("projection_declaration",
RowExclusiveLock);
ScanKeyData key[1];
Expand All @@ -4338,7 +4359,8 @@ PgColumnarDeleteProjectionDeclarationsForRel(Oid relid)
ScanKeyInit(&key[0], Anum_projection_declaration_rel, BTEqualStrategyNumber,
F_OIDEQ, ObjectIdGetDatum(relid));

scan = systable_beginscan(rel, InvalidOid, false, NULL, 1, key);
scanIdx = pgcolumnar_scan_index_oid(rel, "projection_declaration_pkey");
scan = systable_beginscan(rel, scanIdx, OidIsValid(scanIdx), NULL, 1, key);
while (HeapTupleIsValid(tuple = systable_getnext(scan)))
CatalogTupleDelete(rel, &tuple->t_self);
systable_endscan(scan);
Expand Down
Loading
Loading