diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cf16dc30a..5ab335621 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -309,8 +309,11 @@ jobs: # developer box without it; in CI that would be green forever, so CI proves # the module imports and sets PGC_REQUIRE, which turns a decline into a # failure (OffgridwithJD, review of #1286). + # DUCKDB TOO (#1341): the one reader of an exported Parquet file that is not + # pyarrow, pinned beside it in requirements-test.txt and required the same way. /tmp/pgcvenv/bin/python -c 'import pyarrow, pyarrow.parquet; print("pyarrow", pyarrow.__version__)' - export PGC_REQUIRE=pyarrow + /tmp/pgcvenv/bin/python -c 'import duckdb; print("duckdb", duckdb.__version__)' + export PGC_REQUIRE=pyarrow,duckdb PYTHONPATH=. /tmp/pgcvenv/bin/pytest -q \ --pg-config /usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config \ --pgc-expect-tests "$WANT" \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d1baa30c..115dc8bfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -236,6 +236,19 @@ true until the next version shipped. over a failed attempt's file is allowed. ### Changed +- **A Parquet timestamp read into the other PostgreSQL timestamp type is + converted in the session's time zone.** The reader ignored the file's + `isAdjustedToUTC` flag. A wall-time column read into `timestamptz` was taken as + UTC (12:00 read as 08:00-04 in a New York session), and a UTC-adjusted column + read into `timestamp` kept its UTC wall time. Both now follow PostgreSQL's casts. + Same-type reads are unchanged, and so is any read in a UTC session (#1706). + A legacy `TIMESTAMP_MILLIS` or `TIMESTAMP_MICROS` column with no logical type, + as pgcolumnar's own exporter wrote before #1707, is read as UTC-adjusted, as + the Parquet format defines it. The Iceberg foreign table no longer prunes on a + `year()`, `month()`, `day()` or `hour()` partition whose source column is the + other timestamp type: its partition values are in the source's type, so they + no longer bound the converted rows. + `test/parquet_timestamp_zone.sh`, `test/iceberg_fdw.sh` and their twins. - `GROUP BY` is vectorized by default (#1292). `pgcolumnar.enable_group_vectorization` is now `on`. The planner takes the @@ -431,6 +444,30 @@ true until the next version shipped. ended the cleanup early: the segments stayed pinned until the server restarted, and the second cancel was reported instead of the first. The cleanup now finishes whatever arrives, in the time a single cancel already took (#1745). +- **Parquet row-group skipping now uses a constant of another integer or float + type.** A bare integer literal is `int4`, so `WHERE id > 150000` on a `bigint` + column of a Parquet foreign table read every row group; only `150000::int8` + skipped. The check now compares through the operator family's cross-type + comparison, as the native columnar scan already did. Answers were always correct; + this only reads less (#1712). A `timestamp` column compared with a `timestamptz` + constant still does not skip: across a spring-forward gap the session zone's + conversion does not keep the statistics' order. `test/parquet_skip_cross_type.sh` + and `test_parquet_skip_cross_type.py`. +- **A cancelled `parallel_export_parquet` to `s3://` no longer leaves its multipart + uploads open.** A worker learns its upload id only from the answer to + CreateMultipartUpload, and the store creates the upload when the request + arrives. A worker that ended while it waited for that answer left an upload + open on the store that nothing could abort, billed until a lifecycle rule + removed it. The dispatcher's cleanup now lists the uploads under the export's + prefix and aborts each one whose key is one of the export's own part names. + Uploads under the same prefix with any other key are left alone. The aborts run + inside the cleanup that #1741 made safe from a second cancel, so a second cancel + during the cleanup does not skip them. The cleanup holds interrupts only while + it waits, and those waits are bounded: 5 s for the workers to stop, then 10 s + for the listing and the aborts. The object-store module gains two entries for + this, `list_uploads` and `abort_upload` (module ABI 6), so the module and the + extension must come from the same build (#1743, #1722). + `test/objstore_export_open_uploads.sh` and `test_objstore_export_open_uploads.py`. - **`export_parquet` no longer marks a `timestamp` column as a UTC instant.** Both `timestamp` and `timestamptz` were written as the converted type `TIMESTAMP_MICROS`, which the Parquet spec defines as adjusted to UTC. Every other reader (pyarrow, diff --git a/docs/limitations.md b/docs/limitations.md index d64ba6fe0..243816e63 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -1091,9 +1091,14 @@ correct rows; these are the conditions under which it can skip at all: bytea, uuid, and boolean columns are filtered but never skipped, whatever their statistics. A `numeric` column follows its storage: one written as an INT32 or INT64 DECIMAL does skip, one written as a byte-array DECIMAL does not. -- The constant's type must match the column's type exactly. A cross-type - comparison such as `ts >= DATE '2026-01-01'` against a `timestamp` column, or - `bigint_col > 5::int`, does not skip. +- The constant may be of another type in the operator's family. So + `bigint_col > 5` skips, though `5` is an `int4`. So do `real_col < 0.4` and + `ts >= DATE '2026-01-01'` on a `timestamp` column. A comparison that casts the + column instead does not skip. An example is `bigint_col < 5.0`, where `5.0` is a + `numeric`. A `timestamp` column compared with a `timestamptz` constant does not + skip either. The comparison turns each wall time into an instant in the session's + time zone. Across a spring-forward gap that does not keep their order, so the + statistics would not bound the rows. - The row group's statistics must carry both a minimum and a maximum, and the interval must not be inverted. Two cases decode to an interval that the reader does not trust for skipping. The first is an unsigned Parquet column that diff --git a/docs/user-guide.md b/docs/user-guide.md index cf8252056..e78b40614 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -245,6 +245,24 @@ EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events_parquet WHERE ts >= '2026-01- -- Files: 4 ``` +**Timestamps and time zones.** A Parquet timestamp column holds either instants +(UTC-adjusted) or wall times with no zone. Read into the matching type +(`timestamptz` for instants, `timestamp` for wall times), a value comes back +unchanged. Read into the other type, it is converted as PostgreSQL's own cast +converts it, in the session's `TimeZone`: + +- a wall time read into `timestamptz` is taken as local time in that zone, like + `'2024-06-01 12:00'::timestamp::timestamptz`; +- an instant read into `timestamp` becomes the local wall time, like + `'2024-06-01 12:00+00'::timestamptz::timestamp`. + +A foreign-table scan does not skip row groups of a column read this way, because +the conversion does not keep order across a daylight-saving change. For the same +reason the Iceberg foreign table does not prune files on a `year()`, `month()`, +`day()` or `hour()` partition of such a column. A column marked only with the +older `TIMESTAMP_MILLIS` or `TIMESTAMP_MICROS` type, with no `isAdjustedToUTC` +flag, holds instants: the Parquet format defines it so. + The `path` may also be an object-storage URL. Every read and export function, and both foreign-data wrappers, accept an `s3://`, `http://`, or `https://` URL where they accept a local path. diff --git a/objstore/columnar_objstore_module.c b/objstore/columnar_objstore_module.c index 2244cb8e9..1fd6db1c2 100644 --- a/objstore/columnar_objstore_module.c +++ b/objstore/columnar_objstore_module.c @@ -65,6 +65,7 @@ #include "utils/guc.h" #include "utils/memutils.h" #include "utils/resowner.h" +#include "utils/timestamp.h" #include "utils/wait_event.h" #include "columnar_objstore.h" @@ -122,6 +123,13 @@ struct PgColumnarObjHandle SSL *ssl; #endif + /* + * When not 0, every wait on this handle ends by this time, and the request + * raises once it passes (#1743). Set by the calls that take a timeout. + */ + TimestampTz deadline; + int deadline_ms; /* the timeout it came from, for the message */ + /* receive staging */ uint8 *rb; int rb_off; /* consumed up to here */ @@ -240,24 +248,48 @@ os_resource_release(ResourceReleasePhase phase, bool isCommit, /* ------------------------------------------------------------------ waiting */ +static void +os_deadline_passed(PgColumnarObjHandle *h) +{ + ereport(ERROR, + (errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("columnar: \"%s\" did not answer within %d ms", + h->url, h->deadline_ms))); +} + static void os_wait(PgColumnarObjHandle *h, uint32 sockEvents, int *quiet) { int rc; + uint32 events = WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH | sockEvents; + long timeout = OS_WAIT_SLICE_MS; /* * WL_TIMEOUT, or the slice is not a slice: without it WaitLatchOrSocket * ignores the timeout and waits for the socket or the latch alone, so a * silent peer never counts a quiet slice. Under proc_exit (a terminated * worker aborting its upload, #1727) interrupts are held and the latch - * alone never ends the wait either. + * alone never ends the wait either. A handle with a deadline (#1743) has + * each slice cut to the time left, and the deadline is checked before and + * after the wait, so it is bounded whether or not interrupts are held. */ - rc = WaitLatchOrSocket(MyLatch, - WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH | sockEvents, - h->fd, OS_WAIT_SLICE_MS, PG_WAIT_EXTENSION); + if (h->deadline != 0) + { + long left = TimestampDifferenceMilliseconds(GetCurrentTimestamp(), + h->deadline); + + if (left <= 0) + os_deadline_passed(h); + timeout = Min(timeout, left); + } + rc = WaitLatchOrSocket(MyLatch, events, h->fd, timeout, PG_WAIT_EXTENSION); if (rc & WL_LATCH_SET) ResetLatch(MyLatch); CHECK_FOR_INTERRUPTS(); + if (h->deadline != 0 && + !(rc & (WL_SOCKET_READABLE | WL_SOCKET_WRITEABLE)) && + GetCurrentTimestamp() >= h->deadline) + os_deadline_passed(h); if (rc & (WL_SOCKET_READABLE | WL_SOCKET_WRITEABLE)) *quiet = 0; @@ -2664,6 +2696,237 @@ objstore_list_objects(const char *url, const PgColumnarObjStoreConfig *cfg, return out; } +/* --------------------------------------------- ListMultipartUploads (#1743) + * + * The response is endpoint XML like a ListObjectsV2 page, read with the same + * bounded extractor. Each element carries a and an ; + * a truncated page names the next page by NextKeyMarker and + * NextUploadIdMarker. + */ +static void +os_uploads_parse_page(const char *body, int64 blen, const char *bucket, + List **keys, List **ids, bool *truncated, + char **nextKey, char **nextId) +{ + StringInfoData v; + StringInfoData kbuf; + int64 pos = 0; + + *truncated = false; + *nextKey = NULL; + *nextId = NULL; + + initStringInfo(&v); + if (os_xml_element(body, blen, "", "", &v)) + *truncated = (v.len == 4 && pg_strncasecmp(v.data, "true", 4) == 0); + resetStringInfo(&v); + if (os_xml_element(body, blen, "", "", &v) && + v.len > 0) + *nextKey = pstrdup(v.data); + resetStringInfo(&v); + if (os_xml_element(body, blen, "", + "", &v)) + *nextId = pstrdup(v.data); + + initStringInfo(&kbuf); + for (;;) + { + int64 us = os_xml_find(body, blen, pos, ""); + int64 ue; + + if (us < 0) + break; + ue = os_xml_find(body, blen, us, ""); + if (ue < 0) + break; + resetStringInfo(&kbuf); + appendStringInfo(&kbuf, "s3://%s/", bucket); + resetStringInfo(&v); + if (os_xml_element(body + us, ue - us, "", "", &kbuf) && + os_xml_element(body + us, ue - us, "", "", &v) && + v.len > 0) + { + *keys = lappend(*keys, pstrdup(kbuf.data)); + *ids = lappend(*ids, pstrdup(v.data)); + } + pos = ue; + if (list_length(*keys) > OS_LIST_MAX_KEYS) + ereport(ERROR, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("columnar: upload listing exceeded %d uploads", + OS_LIST_MAX_KEYS))); + } + pfree(kbuf.data); + pfree(v.data); +} + +static void +os_set_deadline(PgColumnarObjHandle *h, int timeout_ms) +{ + if (timeout_ms > 0) + { + h->deadline = TimestampTzPlusMilliseconds(GetCurrentTimestamp(), + timeout_ms); + h->deadline_ms = timeout_ms; + } +} + +static char ** +objstore_list_uploads(const char *url, const PgColumnarObjStoreConfig *cfg, + int timeout_ms, char ***upload_ids, int *nuploads) +{ + const char *bstart; + const char *slash; + char *bucket; + char *encPrefix; + char *keyMarker = NULL; + char *idMarker = NULL; + PgColumnarObjHandle *h; + List *keys = NIL; + List *ids = NIL; + char **outKeys; + char **outIds; + int pages = 0; + int i; + + if (pg_strncasecmp(url, "s3://", 5) != 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("columnar: listing uploads is only supported for s3:// URLs, not \"%s\"", + url))); + bstart = url + 5; + slash = strchr(bstart, '/'); + if (slash == NULL || slash == bstart) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("columnar: \"%s\" is not s3://bucket/prefix", url))); + bucket = pnstrdup(bstart, slash - bstart); + encPrefix = os_uriencode_query(slash + 1); + + /* as objstore_list_objects: resolve from a dummy key, request the bucket */ + h = os_write_handle(psprintf("s3://%s/_", bucket), cfg); + os_set_deadline(h, timeout_ms); + { + MemoryContext oldcxt = MemoryContextSwitchTo(TopMemoryContext); + StringInfoData bp; + + initStringInfo(&bp); + appendStringInfoChar(&bp, '/'); + os_uriencode_path(&bp, bucket); + if (h->abspath != NULL) + pfree(h->abspath); + h->abspath = bp.data; + MemoryContextSwitchTo(oldcxt); + } + + PG_TRY(); + { + for (;;) + { + StringInfoData q; + char *body; + int64 blen = 0; + bool truncated = false; + char *nextKey = NULL; + char *nextId = NULL; + int before = list_length(keys); + + if (++pages > OS_LIST_MAX_PAGES) + ereport(ERROR, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("columnar: upload listing of \"%s\" exceeded %d pages", + url, OS_LIST_MAX_PAGES))); + + /* canonical query: key-marker < prefix < upload-id-marker < uploads */ + initStringInfo(&q); + if (keyMarker != NULL) + { + char *enc = os_uriencode_query(keyMarker); + + appendStringInfo(&q, "key-marker=%s&", enc); + pfree(enc); + } + appendStringInfo(&q, "prefix=%s&", encPrefix); + if (keyMarker != NULL && idMarker != NULL) + { + char *enc = os_uriencode_query(idMarker); + + appendStringInfo(&q, "upload-id-marker=%s&", enc); + pfree(enc); + } + appendStringInfoString(&q, "uploads="); + + body = os_list_request(h, q.data, &blen); + pfree(q.data); + os_uploads_parse_page(body, blen, bucket, &keys, &ids, + &truncated, &nextKey, &nextId); + pfree(body); + + if (!truncated || nextKey == NULL) + break; + + /* + * A truncated page must add uploads and move the markers on, or a + * broken or hostile endpoint could keep the loop going (the same + * two guards as objstore_list_objects). + */ + if (list_length(keys) == before || + (keyMarker != NULL && strcmp(nextKey, keyMarker) == 0 && + strcmp(nextId ? nextId : "", idMarker ? idMarker : "") == 0)) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg("columnar: upload listing of \"%s\" did not advance", + url))); + keyMarker = nextKey; + idMarker = nextId; + } + } + PG_CATCH(); + { + os_free_handle(h); + PG_RE_THROW(); + } + PG_END_TRY(); + os_free_handle(h); + + *nuploads = list_length(keys); + outKeys = (char **) palloc(sizeof(char *) * Max(*nuploads, 1)); + outIds = (char **) palloc(sizeof(char *) * Max(*nuploads, 1)); + for (i = 0; i < *nuploads; i++) + { + outKeys[i] = (char *) list_nth(keys, i); + outIds[i] = (char *) list_nth(ids, i); + } + *upload_ids = outIds; + return outKeys; +} + +static int +objstore_abort_upload(const char *url, const char *upload_id, + const PgColumnarObjStoreConfig *cfg, int timeout_ms) +{ + PgColumnarObjHandle *h = os_write_handle(url, cfg); + OsResponse resp; + char *encId = os_uriencode_query(upload_id); + char *query = psprintf("uploadId=%s", encId); + + os_set_deadline(h, timeout_ms); + PG_TRY(); + { + os_write_request(h, "DELETE", query, NULL, 0, &resp, NULL); + } + PG_CATCH(); + { + os_free_handle(h); + PG_RE_THROW(); + } + PG_END_TRY(); + os_free_handle(h); + pfree(query); + pfree(encId); + return resp.status; +} + static void objstore_delete_object(const char *url, const PgColumnarObjStoreConfig *cfg) { @@ -2819,6 +3082,8 @@ static const PgColumnarObjStoreApi objstore_api = { .delete_object = objstore_delete_object, .list_objects = objstore_list_objects, .http_request = objstore_http_request, + .list_uploads = objstore_list_uploads, + .abort_upload = objstore_abort_upload, }; const PgColumnarObjStoreApi * diff --git a/src/columnar_iceberg.c b/src/columnar_iceberg.c index c7deae891..dd4b2732b 100644 --- a/src/columnar_iceberg.c +++ b/src/columnar_iceberg.c @@ -1116,6 +1116,50 @@ ice_field_id_by_name(JsonbContainer *fieldsc, uint32 nf, const char *nm) return -1; } +/* + * Whether the current-schema field named `nm` is a timestamp of the OTHER kind + * than the PostgreSQL column `typid` (timestamp vs timestamptz, any unit). The + * reader converts such a column's rows as PostgreSQL's casts do, in the session + * zone (#1706), but the writer computed its temporal partition cells in the + * source's own kind, so those cells do not bound the converted rows. + */ +static bool +ice_field_timestamp_kind_differs(JsonbContainer *fieldsc, uint32 nf, + const char *nm, Oid typid) +{ + uint32 j; + + if (typid != TIMESTAMPOID && typid != TIMESTAMPTZOID) + return false; + for (j = 0; j < nf; j++) + { + JsonbValue *sf = getIthJsonbValueFromContainer(fieldsc, j); + JsonbValue *v; + bool src_tz; + + if (sf == NULL || sf->type != jbvBinary) + continue; + v = ice_field(sf->val.binary.data, "name"); + if (v == NULL || v->type != jbvString || + (int) strlen(nm) != v->val.string.len || + strncmp(nm, v->val.string.val, v->val.string.len) != 0) + continue; + v = ice_field(sf->val.binary.data, "type"); + if (v == NULL || v->type != jbvString) + return false; + if (v->val.string.len >= 11 && + strncmp(v->val.string.val, "timestamptz", 11) == 0) + src_tz = true; + else if (v->val.string.len >= 9 && + strncmp(v->val.string.val, "timestamp", 9) == 0) + src_tz = false; + else + return false; + return src_tz != (typid == TIMESTAMPTZOID); + } + return false; +} + /* * Map the current partition spec's IDENTITY fields to foreign-table columns, for * FDW pruning. On return *out_attno[k] is the 1-based attno of the table column @@ -1581,7 +1625,10 @@ PgColumnarIcebergDayMap(const char *path, const PgColumnarObjStoreConfig *cfg, * default spec's partition-cell positions and 1-based source attnos for the * fields carrying that transform. The FDW routes each mapped field by its source * column type: the coarse temporal compiler handles the timestamp cases, the - * exact day() compiler handles a day() on a date. + * exact day() compiler handles a day() on a date. A field whose source is a + * timestamp of the other kind than its column (timestamp vs timestamptz) is left + * out, so nothing prunes on it: its cells are in the source's kind, its rows are + * converted in the session zone (#1706). */ void PgColumnarIcebergTemporalMap(const char *path, const PgColumnarObjStoreConfig *cfg, @@ -1676,6 +1723,11 @@ PgColumnarIcebergTemporalMap(const char *path, const PgColumnarObjStoreConfig *c continue; if (ice_field_id_by_name(fieldsc, nf, NameStr(att->attname)) == srcid) { + /* the cells are in the source's kind, the rows in the column's */ + if (ice_field_timestamp_kind_differs(fieldsc, nf, + NameStr(att->attname), + att->atttypid)) + break; pos[cnt] = (int) k; attno[cnt] = a + 1; cnt++; diff --git a/src/columnar_objstore.h b/src/columnar_objstore.h index 632d03dfe..08af5ad32 100644 --- a/src/columnar_objstore.h +++ b/src/columnar_objstore.h @@ -33,7 +33,7 @@ #include "postgres.h" -#define PGCOLUMNAR_OBJSTORE_ABI 5 +#define PGCOLUMNAR_OBJSTORE_ABI 6 /* An open remote object (read) / upload (write). Module owns both. */ typedef struct PgColumnarObjHandle PgColumnarObjHandle; @@ -158,6 +158,36 @@ typedef struct PgColumnarObjStoreApi int nheaders, const char *body, int64 body_len, int64 max_response); + + /* + * Multipart uploads in progress (#1743, ABI v6). A worker learns its + * upload id only from the answer to CreateMultipartUpload, so a worker that + * ends while waiting for that answer leaves an upload nobody holds the id + * of. The parallel export's dispatcher finds those by listing. + * + * list_uploads lists the uploads in progress under `url`, an + * s3://bucket/prefix (ListMultipartUploads, paged by the module). It + * returns a palloc'd array of `*nuploads` keys, each a full s3://bucket/key + * URL, and sets `*upload_ids` to a parallel array of upload ids, both in + * the current memory context. It raises on a transport or parse failure. + * + * abort_upload aborts one upload (AbortMultipartUpload, DELETE ?uploadId=) + * and returns the HTTP status; a status is not an error. It raises on a + * transport failure. + * + * Both take `timeout_ms`: when it is above 0, every wait in the call ends by + * then, and the call raises once that much time has passed, whether or not + * interrupts are held. 0 means the module's ordinary waits. cfg may be NULL. + */ + char **(*list_uploads) (const char *url, + const PgColumnarObjStoreConfig *cfg, + int timeout_ms, + char ***upload_ids, + int *nuploads); + int (*abort_upload) (const char *url, + const char *upload_id, + const PgColumnarObjStoreConfig *cfg, + int timeout_ms); } PgColumnarObjStoreApi; /* diff --git a/src/columnar_parallel_export.c b/src/columnar_parallel_export.c index fe8e846e0..3d2925ead 100644 --- a/src/columnar_parallel_export.c +++ b/src/columnar_parallel_export.c @@ -66,6 +66,7 @@ #include "utils/memutils.h" #include "utils/rel.h" #include "utils/snapmgr.h" +#include "utils/timestamp.h" #define PEXPORT_MAGIC 0x50455850 /* 'PEXP' */ #define PEXPORT_KEY_HEADER 0 @@ -291,11 +292,9 @@ pexport_remove_outputs(const char *dir, int nkeys) * Remote (#623): the dispatcher knows its own key set exactly, so no * listing is needed. The keys are dir/part-0000.parquet through * dir/part-(nkeys-1).parquet; delete each through the module, best-effort - * like the local unlink. A worker the dispatcher TERMINATES dies FATAL - * with its multipart upload neither completed nor aborted, and the ABI's - * delete_object removes a completed key, not an in-progress upload, so an - * orphaned upload is left to the bucket's incomplete-multipart lifecycle - * rule (the same residue #622 documented for a single object). + * like the local unlink. delete_object removes a completed key, not an + * upload in progress: a worker that ended FATAL leaves its upload open, + * and pexport_abort_own_uploads aborts those (#1743). */ if (PgColumnarPathIsRemote(dir)) { @@ -424,6 +423,131 @@ pexport_write_success_marker(const char *dir) PG_END_TRY(); } +/* + * How long the dispatcher's upload cleanup may take (#1743): first for the + * terminated workers to stop, then for the listing and the aborts together. + * Interrupts are held across both, so both are bounded. + */ +#define PEXPORT_STOP_WAIT_MS 5000 +#define PEXPORT_UPLOAD_CLEANUP_MS 10000 + +/* + * pexport_abort_own_uploads + * Abort every multipart upload this export left open on a remote prefix + * (#1743). A worker learns its upload id only from the answer to + * CreateMultipartUpload, and the store creates the upload when the request + * arrives. A worker that ends while it waits for that answer (a cancel + * terminates it, or it is terminated directly) leaves an upload nobody + * holds the id of, and a worker that ends FATAL never runs its own abort. + * So the dispatcher lists the uploads under the prefix and aborts each one + * whose key is one of this export's part names, dir/part-0000.parquet to + * dir/part-(nkeys-1).parquet, exactly. Any other key under the prefix is + * someone else's upload and is left alone. + * + * The workers must have stopped first, so none starts an upload after the + * listing. Interrupts are held from the wait to the last abort, and every + * wait is bounded: the worker wait by PEXPORT_STOP_WAIT_MS on a timed + * latch, the store's answers by the module's timeout_ms, which ends each + * wait at PEXPORT_UPLOAD_CLEANUP_MS after the listing starts. + * + * Best effort, and never raises: anything the listing or an abort raises + * ends the cleanup and is discarded. The caller must have an empty error + * stack, so discarding it discards nothing else. + */ +static void +pexport_abort_own_uploads(const char *dir, int nkeys, + BackgroundWorkerHandle **handles, int nhandles) +{ + const PgColumnarObjStoreApi *api; + TimestampTz start; + MemoryContext cxt = CurrentMemoryContext; + uint32 held; + uint32 cancelHeld; + + if (dir == NULL || !PgColumnarPathIsRemote(dir) || nkeys <= 0) + return; + api = PgColumnarObjStoreGet(); + if (api == NULL || api->list_uploads == NULL || api->abort_upload == NULL) + return; + + HOLD_INTERRUPTS(); + held = InterruptHoldoffCount; + cancelHeld = QueryCancelHoldoffCount; + + /* wait, bounded, for the terminated workers to stop */ + start = GetCurrentTimestamp(); + for (;;) + { + int running = 0; + int i; + + for (i = 0; i < nhandles; i++) + { + pid_t pid; + + if (handles != NULL && handles[i] != NULL && + GetBackgroundWorkerPid(handles[i], &pid) != BGWH_STOPPED) + running++; + } + if (running == 0 || + TimestampDifferenceExceeds(start, GetCurrentTimestamp(), + PEXPORT_STOP_WAIT_MS)) + break; + (void) WaitLatch(MyLatch, WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 100L, PG_WAIT_EXTENSION); + ResetLatch(MyLatch); + } + + start = GetCurrentTimestamp(); + PG_TRY(); + { + char prefix[MAXPGPATH]; + char own[MAXPGPATH]; + char **keys; + char **ids = NULL; + int n = 0; + int i; + + snprintf(prefix, sizeof(prefix), "%s/", dir); + keys = api->list_uploads(prefix, NULL, PEXPORT_UPLOAD_CLEANUP_MS, + &ids, &n); + for (i = 0; i < n; i++) + { + const char *base = keys[i] + strlen(prefix); + int idx; + long left; + + /* this export's own part name, character for character */ + if (strncmp(keys[i], prefix, strlen(prefix)) != 0 || + !pexport_part_index(base, &idx) || idx < 0 || idx >= nkeys) + continue; + snprintf(own, sizeof(own), "%s/part-%04d.parquet", dir, idx); + if (strcmp(keys[i], own) != 0) + continue; + + left = PEXPORT_UPLOAD_CLEANUP_MS - + TimestampDifferenceMilliseconds(start, GetCurrentTimestamp()); + if (left <= 0) + break; + (void) api->abort_upload(keys[i], ids[i], NULL, (int) left); + } + } + PG_CATCH(); + { + /* + * errfinish zeroed the holdoff counts before the longjmp; put back what + * this function held, so the RESUME below balances the HOLD above. + */ + InterruptHoldoffCount = held; + QueryCancelHoldoffCount = cancelHeld; + MemoryContextSwitchTo(cxt); + FlushErrorState(); /* the caller's stack was empty: only ours */ + } + PG_END_TRY(); + + RESUME_INTERRUPTS(); +} + /* error-cleanup: stop every worker so a dispatcher FATAL cannot leave them * reading after the exported snapshot is gone, and remove any partial output */ static void @@ -438,11 +562,19 @@ pexport_cleanup(int code, Datum arg) pexport_remove_outputs(s->dirpath, s->nkeys); } -/* the ERROR path's form of pexport_cleanup, for PgColumnarCleanupAndRethrow */ +/* + * The ERROR path's form of pexport_cleanup, for PgColumnarCleanupAndRethrow, + * which has copied the handled error out and emptied the error stack. On a + * remote prefix, the uploads the workers left open are then aborted (#1743): + * a second cancel that lands in a part DELETE cuts short only that DELETE. + */ static void pexport_cleanup_in_catch(void *arg) { - pexport_cleanup(0, PointerGetDatum(arg)); + PexportSpawn *s = (PexportSpawn *) arg; + + pexport_cleanup(0, PointerGetDatum(s)); + pexport_abort_own_uploads(s->dirpath, s->nkeys, s->handles, s->n); } /* @@ -828,7 +960,9 @@ pgcolumnar_parallel_export_parquet(PG_FUNCTION_ARGS) * macro's PG_CATCH (#1741): the cleanup's remote DELETEs wait on the store, * a second cancel lands there, and swallowing it with the cancel being * handled still on the error stack lost that error ("errstart was not - * called"). + * called"). On a remote prefix that cleanup then aborts the uploads the + * workers left open (#1743), so a second cancel cannot skip the aborts + * either. */ before_shmem_exit(pexport_cleanup, PointerGetDatum(&spawn)); PG_TRY(); @@ -892,6 +1026,8 @@ pgcolumnar_parallel_export_parquet(PG_FUNCTION_ARGS) /* all workers have stopped; drop any part files they wrote so the failed * run leaves a clean directory and a retry is not half-read or blocked */ pexport_remove_outputs(dir, single_table ? workers : npart); + pexport_abort_own_uploads(dir, single_table ? workers : npart, + handles, workers); dsm_detach(seg); if (pushed) PopActiveSnapshot(); diff --git a/src/columnar_parquet_reader.c b/src/columnar_parquet_reader.c index 42d8f15fc..a47601d97 100644 --- a/src/columnar_parquet_reader.c +++ b/src/columnar_parquet_reader.c @@ -34,6 +34,7 @@ #include "fmgr.h" #include "funcapi.h" +#include "access/nbtree.h" #include "access/htup_details.h" #include "common/int.h" #include "access/relation.h" @@ -76,6 +77,8 @@ #include "utils/rel.h" #include "utils/snapmgr.h" #include "utils/timestamp.h" +#include "utils/datetime.h" +#include "pgtime.h" #include "utils/varlena.h" #include "utils/tuplestore.h" #include "utils/typcache.h" @@ -137,6 +140,7 @@ typedef struct PqSchemaCol int converted_type; /* -1 if none */ int time_unit; /* PQ_TU_* for a TIME/TIMESTAMP column */ bool is_timestamp; /* the unit belongs to TIMESTAMP, not TIME */ + bool adjusted_utc; /* TIMESTAMP isAdjustedToUTC: an instant, not a wall time */ int type_length; /* FIXED_LEN_BYTE_ARRAY length */ int scale; /* DECIMAL scale (0 if none) */ int precision; /* DECIMAL precision (0 if none) */ @@ -416,8 +420,12 @@ parse_logical_type(TCReader *r, PqSchemaCol *sc) PgColumnarThriftField(r, &uft, &ufid, &uLast); } } + else if (ifid == 1 && isTs && + (ift == TC_BOOL_TRUE || ift == TC_BOOL_FALSE)) + /* isAdjustedToUTC: compact protocol carries a bool in its type */ + sc->adjusted_utc = (ift == TC_BOOL_TRUE); else - PgColumnarThriftSkip(r, ift); /* isAdjustedToUTC and anything later */ + PgColumnarThriftSkip(r, ift); /* anything later */ } } else @@ -436,6 +444,8 @@ parse_schema_element(TCReader *r, PqSchemaCol *sc) sc->converted_type = -1; sc->time_unit = PQ_TU_NONE; sc->is_timestamp = false; + /* the legacy TIMESTAMP_* converted types are defined as UTC-adjusted */ + sc->adjusted_utc = true; sc->type_length = 0; sc->scale = 0; sc->precision = 0; @@ -875,8 +885,21 @@ typedef struct PqColPlan int type_length; /* FIXED_LEN_BYTE_ARRAY width, for uuid/decimal */ int dec_scale; /* DECIMAL scale, when the target is numeric */ bool is_decimal; /* the bytes are a DECIMAL unscaled integer */ + int ts_shift; /* PQ_TS_*: a cast between wall time and instant */ } PqColPlan; +/* + * A Parquet timestamp read into the OTHER PostgreSQL timestamp type is converted + * as PostgreSQL's own cast would, in the session zone (#1706): a naive (not + * UTC-adjusted) wall time into timestamptz, and an adjusted instant into + * timestamp. The same-kind reads are left as they are. + */ +#define PQ_TS_NONE 0 +#define PQ_TS_WALL_TO_INSTANT 1 +#define PQ_TS_INSTANT_TO_WALL 2 + +static bool pq_ts_shift(int shift, Timestamp in, Timestamp *out); + /* * Convert a stored TIME/TIMESTAMP value to the microseconds PostgreSQL stores, * per the column's declared unit. Returns false if the conversion overflows. @@ -1176,7 +1199,9 @@ plain_value_to_datum(const PqColPlan *plan, const uint8 **p, const uint8 *end, if (!pq_scale_to_usecs(plan->time_unit, v, &us) || pg_sub_s64_overflow(us, PG_TO_UNIX_USECS, &t) || - !IS_VALID_TIMESTAMP((Timestamp) t)) + !IS_VALID_TIMESTAMP((Timestamp) t) || + (plan->ts_shift != PQ_TS_NONE && + !pq_ts_shift(plan->ts_shift, (Timestamp) t, (Timestamp *) &t))) return pq_value_out_of_range(strict, ok, ERRCODE_DATETIME_FIELD_OVERFLOW, tn, v); @@ -2276,6 +2301,47 @@ pq_plan_bind_schema(PqColPlan *plan, const PqSchemaCol *sc) plan->dec_scale = sc->scale; plan->is_decimal = (plan->typid == NUMERICOID && sc->converted_type == PQ_CT_DECIMAL); + plan->ts_shift = PQ_TS_NONE; + if (sc->is_timestamp && !sc->adjusted_utc && plan->typid == TIMESTAMPTZOID) + plan->ts_shift = PQ_TS_WALL_TO_INSTANT; + else if (sc->is_timestamp && sc->adjusted_utc && plan->typid == TIMESTAMPOID) + plan->ts_shift = PQ_TS_INSTANT_TO_WALL; +} + +/* + * The two casts, from the primitives PostgreSQL's timestamp_timestamptz and + * timestamptz_timestamp use, so they behave the same on every major (PG19 renamed + * the exported helper, and PG15-18 export no timestamptz -> timestamp one). + * false when the result is out of range. + */ +static bool +pq_ts_shift(int shift, Timestamp in, Timestamp *out) +{ + struct pg_tm tm; + fsec_t fsec; + int tz; + + if (TIMESTAMP_NOT_FINITE(in)) + { + *out = in; + return true; + } + if (shift == PQ_TS_WALL_TO_INSTANT) + { + if (timestamp2tm(in, NULL, &tm, &fsec, NULL, NULL) != 0) + return false; + tz = DetermineTimeZoneOffset(&tm, session_timezone); + if (tm2timestamp(&tm, fsec, &tz, out) != 0) + return false; + } + else + { + if (timestamp2tm(in, &tz, &tm, &fsec, NULL, NULL) != 0) + return false; + if (tm2timestamp(&tm, fsec, NULL, out) != 0) + return false; + } + return IS_VALID_TIMESTAMP(*out); } /* @@ -4469,9 +4535,11 @@ pqfdw_top_for_attno(ImpTop *tops, int ntops, int attno0) */ static bool pqfdw_clause_excludes_group(ImpLeaf *leaf, PqChunk *ch, int strategy, - bool varLeft, Const *con) + bool varLeft, Const *con, Oid opfamily) { TypeCacheEntry *tce; + FmgrInfo cross; + FmgrInfo *vsConst; const uint8 *p; bool ok; Datum minD; @@ -4484,13 +4552,50 @@ pqfdw_clause_excludes_group(ImpLeaf *leaf, PqChunk *ch, int strategy, if (phys != PQ_INT32 && phys != PQ_INT64 && phys != PQ_FLOAT && phys != PQ_DOUBLE) return false; - if (con->consttype != leaf->plan.typid || con->constisnull) + if (con->constisnull) + return false; + /* + * A wall time is not monotonic in its instant across a spring-forward gap, so + * converted statistics need not bound the converted rows (#1706): never skip. + */ + if (leaf->plan.ts_shift != PQ_TS_NONE) return false; tce = lookup_type_cache(leaf->plan.typid, TYPECACHE_CMP_PROC_FINFO); if (!OidIsValid(tce->cmp_proc_finfo.fn_oid)) return false; + /* + * The statistics are compared with the constant through the operator family's + * comparison function for (column type, constant type), as the native zone-map + * path does (#1712). A bare integer literal is int4, so requiring the column's + * own type left `WHERE id > 150000` on a bigint column skipping nothing. A + * family with no such function, or no family, means no skip. + * + * Except a timestamp column against a timestamptz constant: the comparison + * converts each statistic, a wall time, to an instant in the session zone, + * which is not monotonic across a spring-forward gap. In New York a group + * holding 02:00, 02:30 and 03:00 has max 07:00Z but a row at 07:30Z, so + * `ts > '07:15+00'` would skip it (OffgridwithJD's review of #1737). Never + * skip that pair. A timestamptz column and a timestamp constant is safe: the + * statistics are instants, and only the one constant is converted. + */ + if (leaf->plan.typid == TIMESTAMPOID && con->consttype == TIMESTAMPTZOID) + return false; + if (con->consttype == leaf->plan.typid) + vsConst = &tce->cmp_proc_finfo; + else + { + Oid proc = OidIsValid(opfamily) + ? get_opfamily_proc(opfamily, leaf->plan.typid, con->consttype, BTORDER_PROC) + : InvalidOid; + + if (!OidIsValid(proc)) + return false; + fmgr_info(proc, &cross); + vsConst = ✗ + } + p = ch->stat_min; minD = plain_value_to_datum(&leaf->plan, &p, p + ch->stat_min_len, false, &ok); if (!ok) @@ -4557,9 +4662,12 @@ pqfdw_clause_excludes_group(ImpLeaf *leaf, PqChunk *ch, int strategy, ? (float8) DatumGetFloat4(minD) : DatumGetFloat8(minD); float8 hi = (phys == PQ_FLOAT) ? (float8) DatumGetFloat4(maxD) : DatumGetFloat8(maxD); - float8 c = (phys == PQ_FLOAT) + /* the constant is read by ITS type, which may not be the column's */ + float8 c = (con->consttype == FLOAT4OID) ? (float8) DatumGetFloat4(con->constvalue) - : DatumGetFloat8(con->constvalue); + : (con->consttype == FLOAT8OID) + ? DatumGetFloat8(con->constvalue) + : 0.0; if (isnan(lo) || isnan(hi)) return false; @@ -4570,22 +4678,29 @@ pqfdw_clause_excludes_group(ImpLeaf *leaf, PqChunk *ch, int strategy, return false; } + /* + * The column's value is always the FIRST argument: a cross-type comparison + * function is defined for (column type, constant type) only. + */ +#define PQ_VS(a) \ + DatumGetInt32(FunctionCall2Coll(vsConst, InvalidOid, (a), con->constvalue)) + switch (strategy) { case BTLessStrategyNumber: /* col < const: skip if min >= const */ - return PQ_CMP(minD, con->constvalue) >= 0; + return PQ_VS(minD) >= 0; case BTLessEqualStrategyNumber: /* col <= const: skip if min > const */ - return PQ_CMP(minD, con->constvalue) > 0; + return PQ_VS(minD) > 0; case BTEqualStrategyNumber: /* col = const: skip if const outside [min,max] */ - return PQ_CMP(con->constvalue, minD) < 0 || - PQ_CMP(con->constvalue, maxD) > 0; + return PQ_VS(minD) > 0 || PQ_VS(maxD) < 0; case BTGreaterEqualStrategyNumber: /* col >= const: skip if max < const */ - return PQ_CMP(maxD, con->constvalue) < 0; + return PQ_VS(maxD) < 0; case BTGreaterStrategyNumber: /* col > const: skip if max <= const */ - return PQ_CMP(maxD, con->constvalue) <= 0; + return PQ_VS(maxD) <= 0; default: return false; } +#undef PQ_VS #undef PQ_CMP } @@ -4630,6 +4745,7 @@ pqfdw_compute_skip(ForeignScanState *node, PqFile *pf, List *interp; ListCell *ic; int strategy = 0; + Oid opfamily = InvalidOid; if (!IsA(op, OpExpr) || list_length(op->args) != 2) continue; @@ -4670,6 +4786,7 @@ pqfdw_compute_skip(ForeignScanState *node, PqFile *pf, if (s >= BTLessStrategyNumber && s <= BTGreaterStrategyNumber) { strategy = s; + opfamily = o->opfamily_id; break; } } @@ -4678,7 +4795,7 @@ pqfdw_compute_skip(ForeignScanState *node, PqFile *pf, if (pqfdw_clause_excludes_group(&leaves[top->firstLeaf], &pf->rgs[rg].chunks[top->firstLeaf], - strategy, varLeft, con)) + strategy, varLeft, con, opfamily)) { skipGroup[rg] = true; break; diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e5e80d92d..6533522e0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -5196,6 +5196,36 @@ objstore_export_double_cancel objstore_export_double_cancel double cancel: the s objstore_export_double_cancel objstore_export_double_cancel premise: a worker was uploading before the first cancel 15;16;17;18;19 never - objstore_export_double_cancel objstore_export_double_cancel premise: sigv4 fixture server is up 15;16;17;18;19 never - objstore_export_double_cancel objstore_export_double_cancel premise: the first cancel's cleanup was deleting when the second cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads cancel: no completed object remains at the prefix 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads cancel: no upload of this export is left open 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads never lists the uploads (the call to list_uploads removed) +objstore_export_open_uploads objstore_export_open_uploads cancel: the export reports its own cancel (57014) 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads cancel: the foreign uploads under the prefix are left open 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads aborts every upload listed under the prefix (the own-part-name match removed);src/columnar_parallel_export.c: the own-part match is off by one (idx > nkeys for idx >= nkeys), so part- counts as the export's own +objstore_export_open_uploads objstore_export_open_uploads cancel: the server survived 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads double cancel: no upload of this export is left open 15;16;17;18;19 2026-10-23 src/columnar_parallel_export.c: pexport_abort_own_uploads never lists the uploads (the call to list_uploads removed) +objstore_export_open_uploads objstore_export_open_uploads double cancel: the export reports its own cancel (57014) 15;16;17;18;19 2026-10-23 src/columnar_parallel_export.c: the export's PG_CATCH runs pexport_cleanup_in_catch with the handled error still on the stack, then PG_RE_THROW (PgColumnarCleanupAndRethrow bypassed) +objstore_export_open_uploads objstore_export_open_uploads double cancel: the foreign uploads under the prefix are left open 15;16;17;18;19 2026-10-23 src/columnar_parallel_export.c: pexport_abort_own_uploads aborts every upload listed under the prefix (the own-part-name match removed);src/columnar_parallel_export.c: the own-part match is off by one (idx > nkeys for idx >= nkeys), so part- counts as the export's own +objstore_export_open_uploads objstore_export_open_uploads double cancel: the server log has no 'errstart was not called' 15;16;17;18;19 2026-10-23 src/columnar_parallel_export.c: the export's PG_CATCH runs pexport_cleanup_in_catch with the handled error still on the stack, then PG_RE_THROW (PgColumnarCleanupAndRethrow bypassed) +objstore_export_open_uploads objstore_export_open_uploads double cancel: the server survived 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads parts: no upload of this export is left open 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads parts: the export reports its own cancel (57014) 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads parts: the foreign uploads under the prefix are left open 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads aborts every upload listed under the prefix (the own-part-name match removed);src/columnar_parallel_export.c: the own-part match is off by one (idx > nkeys for idx >= nkeys), so part- counts as the export's own +objstore_export_open_uploads objstore_export_open_uploads parts: the server survived 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: cancel: every worker's CreateMultipartUpload was unanswered when the cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: double cancel: every worker's CreateMultipartUpload was unanswered when the first cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: double cancel: the first cancel's cleanup was deleting a part when the second cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: parts: every worker was sending parts when the cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: sigv4 fixture server is up 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: silent listing: every worker's CreateMultipartUpload was unanswered when the cancel landed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: terminate: every worker's CreateMultipartUpload was unanswered when the workers were terminated 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads premise: the foreign uploads were open before the exports 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads silent listing: the cleanup asked the store for its open uploads 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads never lists the uploads (the call to list_uploads removed) +objstore_export_open_uploads objstore_export_open_uploads silent listing: the export reports its own cancel (57014) 15;16;17;18;19 2026-10-23 src/columnar_parallel_export.c: the export's PG_CATCH runs pexport_cleanup_in_catch with the handled error still on the stack, then PG_RE_THROW (PgColumnarCleanupAndRethrow bypassed) +objstore_export_open_uploads objstore_export_open_uploads silent listing: the export returned within 30 s of the cancel 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: list_uploads called with timeout_ms 0, so the listing's waits are unbounded while interrupts are held +objstore_export_open_uploads objstore_export_open_uploads silent listing: the server survived 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads terminate: no upload of this export is left open 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads never lists the uploads (the call to list_uploads removed) +objstore_export_open_uploads objstore_export_open_uploads terminate: the export reports that a worker failed 15;16;17;18;19 never - +objstore_export_open_uploads objstore_export_open_uploads terminate: the foreign uploads under the prefix are left open 15;16;17;18;19 2026-10-22 src/columnar_parallel_export.c: pexport_abort_own_uploads aborts every upload listed under the prefix (the own-part-name match removed);src/columnar_parallel_export.c: the own-part match is off by one (idx > nkeys for idx >= nkeys), so part- counts as the export's own +objstore_export_open_uploads objstore_export_open_uploads terminate: the server survived 15;16;17;18;19 never - objstore_http_read objstore_http_read A: count(*) over http 15;16;17;18;19 2026-10-22 columnar_objstore_module.c: the Range header's end is one past the last byte wanted objstore_http_read objstore_http_read A: full scan http == local 15;16;17;18;19 2026-10-22 columnar_objstore_module.c: the Range header's end is one past the last byte wanted objstore_http_read objstore_http_read A: projection http == local 15;16;17;18;19 2026-10-22 columnar_objstore_module.c: the Range header's end is one past the last byte wanted @@ -5622,8 +5652,22 @@ parquet_count_bounds parquet_count_bounds premise: the server is up before any m parquet_count_bounds parquet_count_bounds the num_rows rejection names num_rows in its detail 15;16;17;18;19 never - parquet_count_bounds parquet_count_bounds the num_values rejection names num_values in its detail 15;16;17;18;19 never - parquet_count_bounds parquet_count_bounds the server is still up after the honest import 15;16;17;18;19 never - +parquet_export parquet_export duckdb extended row count 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: row-group num_rows written one high +parquet_export parquet_export duckdb non-null count(b) 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: a negative int4 written as null;src/columnar_parquet.c: leaf names written without their first character;src/columnar_parquet.c: row-group num_rows written one high +parquet_export parquet_export duckdb reads decimal 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: DECIMAL written little-endian;src/columnar_parquet.c: leaf names written without their first character +parquet_export parquet_export duckdb row count 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: row-group num_rows written one high +parquet_export parquet_export empty export rows written 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: export_parquet returns its row count one high +parquet_export parquet_export empty parquet readable 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: leaf names written without their first character +parquet_export parquet_export export_parquet rows written 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: export_parquet returns its row count one high +parquet_export parquet_export extended export rows written 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: export_parquet returns its row count one high +parquet_export parquet_export extended parquet types 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: leaf names written without their first character;src/columnar_parquet.c: numeric(p,s) never mapped to DECIMAL +parquet_export parquet_export extended values match heap oracle 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: DECIMAL written little-endian;src/columnar_parquet.c: a date written one day late;src/columnar_parquet.c: leaf names written without their first character;src/columnar_parquet.c: numeric(p,s) never mapped to DECIMAL +parquet_export parquet_export parquet schema 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: int2 written without its INT_16 annotation;src/columnar_parquet.c: leaf names written without their first character +parquet_export parquet_export parquet values match heap oracle 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: a negative int4 written as null;src/columnar_parquet.c: every bool written inverted;src/columnar_parquet.c: leaf names written without their first character +parquet_export parquet_export reject non-columnar table 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: the columnar-relation check skipped +parquet_export parquet_export reject unsupported type 15;16;17;18;19 2026-10-22 src/columnar_parquet.c: the unsupported-type refusal formats the typmod as a type parquet_export_stats parquet_export_stats a bigint predicate skips 15;16;17;18;19 2026-10-21 columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const -parquet_export_stats parquet_export_stats a cross-type constant skips nothing, on the identical predicate 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses a constant of another type +parquet_export_stats parquet_export_stats a cross-type constant skips as the matching one does, on the identical predicate 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing parquet_export_stats parquet_export_stats a date predicate skips 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const parquet_export_stats parquet_export_stats a decimal-point literal casts the column, not the constant 15;16;17;18;19 never - parquet_export_stats parquet_export_stats a float >= predicate above every bound still skips nothing 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses > and >= on a float column @@ -5649,8 +5693,8 @@ parquet_export_stats parquet_export_stats control: the parser reports the deprec parquet_export_stats parquet_export_stats control: the quoted form casts the constant to the column's type 15;16;17;18;19 never - parquet_export_stats parquet_export_stats control: the same column skips when the constant matches its type 15;16;17;18;19 2026-10-21 columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type parquet_export_stats parquet_export_stats control: the unadorned form resolves to the column's type instead 15;16;17;18;19 never - +parquet_export_stats parquet_export_stats digits too wide for an integer are a bigint, and an int4 column skips as well 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing parquet_export_stats parquet_export_stats digits too wide for an integer are a bigint, and skip 15;16;17;18;19 2026-10-21 columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const -parquet_export_stats parquet_export_stats digits too wide for an integer are a bigint, so an int4 column refuses 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses a constant of another type parquet_export_stats parquet_export_stats each bucket holds its own contiguous id range 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats every ColumnOrder is TYPE_ORDER 15;16;17;18;19 2026-10-21 columnar_parquet.c: FileMetaData.column_orders not written parquet_export_stats parquet_export_stats every bound is its physical width 15;16;17;18;19 2026-10-21 columnar_parquet.c: write_statistics writes no min_value or max_value for any type @@ -5759,6 +5803,7 @@ parquet_export_stats parquet_export_stats the FDW result equals the oracle for [ parquet_export_stats parquet_export_stats the FDW result equals the oracle for [d < TIMESTAMP '2000-01-05 00:00:00'] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [d > DATE '2060-01-01'] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [f4 < '0.4'] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds +parquet_export_stats parquet_export_stats the FDW result equals the oracle for [f4 < 0.4] 15;16;17;18;19 never - parquet_export_stats parquet_export_stats the FDW result equals the oracle for [f4 < 1.0::real] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const parquet_export_stats parquet_export_stats the FDW result equals the oracle for [f4 >= 1.0::real] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i2 < '500'] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const @@ -5767,6 +5812,8 @@ parquet_export_stats parquet_export_stats the FDW result equals the oracle for [ parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i8 < '5'] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i8 < 1000000000::bigint] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i8 < 5000000000] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const +parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i8 < 5::int] 15;16;17;18;19 never - +parquet_export_stats parquet_export_stats the FDW result equals the oracle for [i8 < 5] 15;16;17;18;19 never - parquet_export_stats parquet_export_stats the FDW result equals the oracle for [id < -1] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [id < -5000000000] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the FDW result equals the oracle for [id < 5000] 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const @@ -5779,11 +5826,11 @@ parquet_export_stats parquet_export_stats the footer carries one ColumnOrder per parquet_export_stats parquet_export_stats the four groups report four distinct id minima 15;16;17;18;19 2026-10-21 columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type parquet_export_stats parquet_export_stats the oracle holds every exported row 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds parquet_export_stats parquet_export_stats the oracle's buckets are the file's row groups 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds -parquet_export_stats parquet_export_stats the same date adorned as a timestamp does not skip 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds +parquet_export_stats parquet_export_stats the same date adorned as a timestamp skips as well 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing parquet_export_stats parquet_export_stats the same date unadorned is unknown, takes the column's type and skips 15;16;17;18;19 2026-10-21 columnar_parquet.c: leaf_value_representable admits date and timestamp infinities, so they reach the data and the bounds;columnar_parquet.c: pqstats_reset leaves hasBounds set, so each later group's bounds start from the reset 0;columnar_parquet.c: write_statistics writes no min_value or max_value for any type;columnar_parquet_reader.c: col < const skips a group on max >= const instead of min >= const -parquet_export_stats parquet_export_stats the same digits unquoted are an integer, and skip nothing 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses a constant of another type -parquet_export_stats parquet_export_stats the same digits unquoted are an integer, and skip nothing (int2) 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses a constant of another type -parquet_export_stats parquet_export_stats the same literal unquoted is double precision, and skips nothing 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: row-group skipping no longer refuses a constant of another type +parquet_export_stats parquet_export_stats the same digits unquoted are an integer, and skip as well 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing +parquet_export_stats parquet_export_stats the same digits unquoted are an integer, and skip as well (int2) 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing +parquet_export_stats parquet_export_stats the same literal unquoted is double precision, and skips as well 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the exact-type refusal before #1712 restored, so a constant of another type skips nothing parquet_level_width parquet_level_width a non-positive maximum returns before shifting at all 15;16;17;18;19 2026-10-21 columnar_parquet_format.h: pq_bits_for's non-positive guard is commented out parquet_level_width parquet_level_width and both callers reach the shared one 15;16;17;18;19 never - parquet_level_width parquet_level_width and no signed 1 << remains in it 15;16;17;18;19 2026-10-21 columnar_parquet_format.h: pq_bits_for shifts a signed 1 @@ -5804,6 +5851,38 @@ parquet_nested_import parquet_nested_import nested import round-trips (arrays + parquet_nested_import parquet_nested_import pyarrow import rows 15;16;17;18;19 never - parquet_nested_import parquet_nested_import pyarrow nested file matches oracle 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: an empty Parquet list is read back as a NULL array parquet_nested_import parquet_nested_import round-trips after delete 15;16;17;18;19 2026-10-21 columnar_parquet_reader.c: an empty Parquet list is read back as a NULL array +parquet_skip_cross_type parquet_skip_cross_type a constant on the left of the operator is compared the same way 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the comparison function is called with the constant first, not the column's value +parquet_skip_cross_type parquet_skip_cross_type a float4 NaN constant still finds the float8 column's NaN row 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the float NaN check reads the constant as the column's float type +parquet_skip_cross_type parquet_skip_cross_type a float4 constant below every row skips all 4 row groups of the float8 column 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the column type's own comparison function is used for a constant of another type +parquet_skip_cross_type parquet_skip_cross_type a timestamptz constant after a spring-forward gap finds the timestamp column's row the heap finds (>) 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a timestamp column is compared with a timestamptz constant through the cross-type function +parquet_skip_cross_type parquet_skip_cross_type a timestamptz constant after a spring-forward gap finds the timestamp column's row the heap finds (>=) 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a timestamp column is compared with a timestamptz constant through the cross-type function +parquet_skip_cross_type parquet_skip_cross_type a timestamptz constant equal to a row's instant after a spring-forward gap finds that row, as the heap does 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a timestamp column is compared with a timestamptz constant through the cross-type function +parquet_skip_cross_type parquet_skip_cross_type an int4 constant in an equality on the int8 column skips 3 of the 4 row groups 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the comparison function is called with the constant first, not the column's value +parquet_skip_cross_type parquet_skip_cross_type an int4 constant on the int8 column skips 3 of the 4 row groups 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the comparison function is called with the constant first, not the column's value +parquet_skip_cross_type parquet_skip_cross_type and returns the rows the float8 constant returns 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type and returns the rows the heap returns 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type and returns the rows the int8 constant returns 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type control: a timestamp constant on the timestamp column skips 1 of the 2 row groups 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type control: an int8 constant on the int8 column skips 3 of the 4 row groups 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: the comparison function is called with the constant first, not the column's value +parquet_skip_cross_type parquet_skip_cross_type premise: in New York the heap copy finds row 1 for ts > 07:15Z, ts >= 07:15Z and ts = 07:30Z 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type premise: the file holds 4 row groups (pyarrow's own count) 15;16;17;18;19 never - +parquet_skip_cross_type parquet_skip_cross_type premise: the gap file holds 2 row groups (pyarrow's own count) 15;16;17;18;19 never - +parquet_timestamp_zone parquet_timestamp_zone a legacy TIMESTAMP_MICROS file read into timestamp is the session-zone wall time of its instant 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a column with only a legacy TIMESTAMP_* converted type reads as not UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone a legacy TIMESTAMP_MICROS file read into timestamptz keeps its instant 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a column with only a legacy TIMESTAMP_* converted type reads as not UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone a naive file read into timestamptz in Tokyo is its wall time there, as the cast gives 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a naive column read into timestamptz is converted in America/New_York, not the session zone +parquet_timestamp_zone parquet_timestamp_zone a naive file read into timestamptz is its wall time in the session zone 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: isAdjustedToUTC is ignored, so every timestamp column reads as UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone a naive wall time in the fall-back hour read into timestamptz is what the cast gives 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a naive wall time takes the zone offset in force at that wall time read as UTC, not at the wall time itself +parquet_timestamp_zone parquet_timestamp_zone a row group spanning a spring-forward gap is not skipped on its converted min and max 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a converted timestamp column's row groups are skipped on their converted min and max;columnar_parquet_reader.c: a naive wall time takes the zone offset in force at that wall time read as UTC, not at the wall time itself +parquet_timestamp_zone parquet_timestamp_zone an adjusted file read into timestamp is the session-zone wall time of its instant 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: an adjusted column read into timestamp is not converted to the session-zone wall time +parquet_timestamp_zone parquet_timestamp_zone an export and import round trip keeps timestamp and timestamptz in a non-UTC session 15;16;17;18;19 2026-10-22 columnar_parquet.c: main's exporter before #1707, which marks a timestamp column UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone and that is what PostgreSQL's cast from timestamp gives 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: isAdjustedToUTC is ignored, so every timestamp column reads as UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone and that is what PostgreSQL's cast from timestamptz gives 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: an adjusted column read into timestamp is not converted to the session-zone wall time +parquet_timestamp_zone parquet_timestamp_zone and the Tokyo reading is a different instant from the New York one 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a naive column read into timestamptz is converted in America/New_York, not the session zone +parquet_timestamp_zone parquet_timestamp_zone control: a naive file read into timestamp keeps its wall time 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: isAdjustedToUTC is ignored, so every timestamp column reads as UTC-adjusted +parquet_timestamp_zone parquet_timestamp_zone control: an adjusted file read into timestamptz keeps its instant 15;16;17;18;19 never - +parquet_timestamp_zone parquet_timestamp_zone premise: pyarrow wrote the naive file without isAdjustedToUTC and the other with it 15;16;17;18;19 never - +parquet_timestamp_zone parquet_timestamp_zone premise: the legacy fixture carries the TIMESTAMP_MICROS converted type and no logical type 15;16;17;18;19 never - +parquet_timestamp_zone parquet_timestamp_zone the gap file's wall times 02:00, 02:30 and 03:00 become, sorted, the instants 07:00Z, 07:00Z and 07:30Z 15;16;17;18;19 2026-10-22 columnar_parquet_reader.c: a naive wall time takes the zone offset in force at that wall time read as UTC, not at the wall time itself;columnar_parquet_reader.c: isAdjustedToUTC is ignored, so every timestamp column reads as UTC-adjusted partition_lifecycle partition_lifecycle a NULL cutoff is refused, in both overloads 15;16;17;18;19 2026-10-13 partition_drop_plan without its NULL-cutoff check, in both overloads partition_lifecycle partition_lifecycle a call that would create more than 1000 partitions is refused before any DDL 15;16;17;18;19 2026-10-12 partition_create_ahead without the 1000-partition bound partition_lifecycle partition_lifecycle a cutoff before every partition plans nothing 15;16;17;18;19 2026-10-13 the cutoff filter removed: every partition planned @@ -6115,6 +6194,17 @@ phase6 phase6 sum(id) 15;16;17;18;19 never - phase6 phase6 sum(numeric) 15;16;17;18;19 2026-10-22 columnar_encoding.c: an uncompressed varlena decode turns the first 'w' byte of each vector into 'W' phase6 phase6 sum(smallint) 15;16;17;18;19 never - phase6 phase6 vectors formed 15;16;17;18;19 never - +planner_choice_quality planner_choice_quality [range scan with aggregate] forcing produced a different plan 15;16;17;18;19 2026-10-22 src/columnar_customscan.c: the planner ignores pgcolumnar.enable_custom_scan=off +planner_choice_quality planner_choice_quality [range scan with aggregate] the chosen plan is within the bound of the best alternative 15;16;17;18;19 2026-10-22 src/columnar_customscan.c: pgcolumnar_refined_scan_cost prices the custom scan 100x, so the planner picks the index scan;src/columnar_tableam.c: pgcolumnar_index_fetch_tuple raises an error on every fetch;src/columnar_vector.c: PgColumnarBeginGroupAggScan raises an error whenever the grouped aggregate executes +planner_choice_quality planner_choice_quality [range scan with aggregate] the chosen plan ran to completion 15;16;17;18;19 2026-10-22 src/columnar_vector.c: PgColumnarBeginGroupAggScan raises an error whenever the grouped aggregate executes +planner_choice_quality planner_choice_quality [wide range, narrow projection] forcing produced a different plan 15;16;17;18;19 2026-10-22 src/columnar_customscan.c: the planner ignores pgcolumnar.enable_custom_scan=off +planner_choice_quality planner_choice_quality [wide range, narrow projection] the chosen plan is within the bound of the best alternative 15;16;17;18;19 2026-10-22 src/columnar_customscan.c: PgColumnarBeginCustomScan raises an error whenever the scan executes;src/columnar_customscan.c: pgcolumnar_refined_scan_cost prices the custom scan 100x, so the planner picks the index scan;src/columnar_tableam.c: pgcolumnar_index_fetch_tuple raises an error on every fetch +planner_choice_quality planner_choice_quality [wide range, narrow projection] the chosen plan ran to completion 15;16;17;18;19 2026-10-22 src/columnar_customscan.c: PgColumnarBeginCustomScan raises an error whenever the scan executes +planner_choice_quality planner_choice_quality fixture rows 15;16;17;18;19 never - +planner_choice_quality planner_choice_quality premise: [range scan with aggregate] the chosen plan was identified 15;16;17;18;19 never - +planner_choice_quality planner_choice_quality premise: [wide range, narrow projection] the chosen plan was identified 15;16;17;18;19 never - +planner_choice_quality planner_choice_quality premise: the index key is correlated, which is the case that misprices 15;16;17;18;19 never - +planner_choice_quality planner_choice_quality premise: the payload did not compress, so the fetch path is exercised 15;16;17;18;19 never - preimage_rewrite preimage_rewrite ...and prunes as well as the equivalent explicit range 15;16;17;18;19 2026-10-21 columnar_customscan.c: the date_trunc preimage keys are derived and then dropped, so no rewrite happens preimage_rewrite preimage_rewrite ...and still answers correctly with the fold enabled 15;16;17;18;19 never - preimage_rewrite preimage_rewrite ...and the heap agrees it is unsatisfiable 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 8b62c6794..9b3b1e662 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 27 +suites_not_covered 25 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -7056,4 +7056,110 @@ suites_not_covered 27 # never 4520 + 15 = 4535 # dated 3443 + 2 = 3445 # 4535 + 3445 == 7980 -checks_never_observed_red 4535 +# +# #1706 item 2, parquet_timestamp_zone seeded, with its twin (jdatcmd): 4535 -> 4538. A new +# suite, so suites_not_covered does not move: it is registered and covered in the same change. +# It enters with 16 checks from five-major runs (all green on each of 15-19, shell and twin +# alike). 13 are dated by seven mutations, each reddening the same names in both twins: the +# reader ignoring isAdjustedToUTC (6), not converting an adjusted column read into +# timestamp (2), skipping a converted column's row groups on their converted statistics +# (the spring-forward group, 1), main's exporter before #1707 (the round trip, 1), the +# conversion pinned to New York instead of the session zone (the Tokyo pair, 2), a legacy +# TIMESTAMP_MICROS column read as naive (the legacy pair, 2), and the zone offset taken at +# the wall time read as UTC (the fall-back hour, with the two gap checks, 1 new). The two +# premises and the adjusted-into-timestamptz control arrive never. iceberg_fdw's two new +# mixed-kind hour() checks add no rows: that suite is not yet seeded. +# +# ledger rows 7980 + 16 = 7996 +# never 4535 + 3 = 4538 +# dated 3445 + 13 = 3458 +# 4538 + 3458 == 7996 +# +# #1712, parquet_skip_cross_type seeded, with its twin (jdatcmd): 4538 -> 4541. A new suite, +# so suites_not_covered does not move: it is registered and covered in the same change. It +# enters with 9 checks from five-major runs (all green on each of 15-19, shell and twin +# alike). 6 are dated by three mutations, each reddening the same names in both twins: the +# comparison function called with the constant first (4), the column type's own function +# used for a constant of another type (the negative float4 arm, 1), and the NaN check +# reading the constant as the column's type (1). The premise and the two answer controls +# arrive never. +# +# ledger rows 7996 + 9 = 8005 +# never 4538 + 3 = 4541 +# dated 3458 + 6 = 3464 +# 4541 + 3464 == 8005 +# +# #1712, parquet_export_stats follows the new rule (jdatcmd): 4541 -> 4544. Six arms measured +# the refusal of a cross-type constant (it skipped nothing); each is renamed for what it now +# asserts, the skip its same-type twin gets, and keeps its row's history. Three soundness +# arms are added for the predicates that had none, [i8 < 5::int], [i8 < 5] and [f4 < 0.4], +# and arrive never, like the other oracle arms. suites_not_covered does not move. +# +# ledger rows 8005 + 3 = 8008 +# never 4541 + 3 = 4544 +# dated 3464 + 0 = 3464 +# 4544 + 3464 == 8008 +# +# #1712, parquet_skip_cross_type's spring-forward arms, with its twin (jdatcmd): 4544 -> 4548. +# OffgridwithJD's review measured a timestamp column against a timestamptz constant: the +# cross-type comparison converts each wall-time statistic to a session-zone instant, which +# is not monotonic across a spring-forward gap, so in New York `ts > '2024-03-10 07:15+00'` +# skipped the group holding 02:30 (07:30Z). The reader now refuses that pair. 7 checks are +# added, green on each of 15-19 in shell and twin alike. 3 are dated by one mutation, the +# guard deleted, which reddens the same three names in both twins: the >, >= and = arms +# against the heap copy. The two premises, the same-type skip control and its answer +# arrive never. suites_not_covered does not move. +# +# ledger rows 8008 + 7 = 8015 +# never 4544 + 4 = 4548 +# dated 3464 + 3 = 3467 +# 4548 + 3467 == 8015 +# +# ledger rows 8015 + 14 = 8029 +# never 4548 + 0 = 4548 +# dated 3467 + 14 = 3481 +# 4548 + 3481 == 8029 +# +# #1341, planner_choice_quality seeded, with its twin (jdatcmd): 4548 -> 4553, and +# suites_not_covered 26 -> 25. The suite had no rows. It enters with 11 checks from +# five-major runs (all green on each of 15-19, shell and twin alike), after two fixes. +# psql prints "Time:" after a statement that errored, so a chosen plan that raised an +# error in 2 ms was read as a plan that answered in 2 ms: with the custom scan or the +# grouped aggregate raising at execution, main's shell passed every check. The chosen +# plan must now run to completion, by psql's exit status, and a forced alternative that +# fails fails the comparison. And the ratio check's name carried the node that won, so +# it had one name green and another red; the names now carry only the query's label. +# "forcing produced a different plan" lost its premise prefix (#1339): with +# pgcolumnar.enable_custom_scan ignored it is the check that goes red. 6 are dated by +# five mutations, each reddening the same names in both twins. 5 arrive never: fixture +# rows and four premises. The suite stays out of CI's suites job under PGC_SKIP_TIMING; +# its twin declares the two measured checks per query unrunnable under that flag. +# +# ledger rows 8029 + 11 = 8040 +# never 4548 + 5 = 4553 +# dated 3481 + 6 = 3487 +# 4553 + 3487 == 8040 +# +# #1743, a cancelled remote export's open uploads, objstore_export_open_uploads seeded +# with its twin (jdatcmd): 4553 -> 4571. A new suite, registered with its rows, so +# suites_not_covered stays 34. A parallel_export_parquet to s3:// is ended while its +# workers' multipart uploads are open: cancelled while they send parts; cancelled while +# every CreateMultipartUpload is held unanswered by the fixture server's new +# __uploads_delay__ hook; its workers terminated in that window; cancelled there and +# cancelled again while the cleanup's part DELETE is held by #1741's __delete_delay__; +# and cancelled there while the store does not answer the listing. 30 checks: 12 are +# dated by four mutations of the dispatcher's new cleanup (no listing, every listed key +# aborted, the part match off by one, and the cleanup run with the handled error still +# on the stack, which reports "errstart was not called" as #1741 did), each reddening +# the same names in both twins on PG17. 18 arrive never: eight premises, three reported +# errors, the empty prefix, five survival checks, and "parts: no upload of this export +# is left open": on this base #1727's worker exit callback aborts a sending worker's +# upload, so dropping the listing alone no longer reddens it. A fifth mutation, the +# listing's deadline dropped, reddens "silent listing: the export returned within 30 s +# of the cancel" only, which was already dated. +# +# ledger rows 8040 + 30 = 8070 +# never 4553 + 18 = 4571 +# dated 3487 + 12 = 3499 +# 4571 + 3499 == 8070 +checks_never_observed_red 4571 diff --git a/test/fixtures/parquet/legacy_timestamp_micros.parquet b/test/fixtures/parquet/legacy_timestamp_micros.parquet new file mode 100644 index 000000000..8c842e2ff Binary files /dev/null and b/test/fixtures/parquet/legacy_timestamp_micros.parquet differ diff --git a/test/iceberg_fdw.sh b/test/iceberg_fdw.sh index 22f2ca33f..2d41a85c3 100755 --- a/test/iceberg_fdw.sh +++ b/test/iceberg_fdw.sh @@ -353,6 +353,31 @@ if ftmpx yd byyear_d "dt date" && ftmpx mo bymonth_d "dt date" \ "$(oraclex hz "tt >= TIMESTAMPTZ '2021-03-01 01:00:00+00'")" fi +# ---- a temporal transform read into the OTHER timestamp type (#1706) --------- +# The reader converts a naive (timestamp) source read into timestamptz, and a +# timestamptz source read into timestamp, as PostgreSQL's casts do in the session +# zone. The writer computed each partition cell in the source's own kind, so a +# cell does not bound the converted rows, and the FDW must not prune on it. +# In New York, byhour's naive 2021-03-01 00:00, 01:00, 05:00 are 05:00Z, 06:00Z +# and 10:00Z; byhour_tz's 00:00Z, 01:00Z, 05:00Z are the wall times 2021-02-28 +# 19:00, 20:00 and 2021-03-01 00:00. Each FDW read is compared with iceberg_scan +# under the same declaration and zone, and check_text refuses two empty readings. +qny() { PGOPTIONS="-c TimeZone=America/New_York" q "$1"; } +if [ -n "${MD_th:-}" ] && ftmpx hn byhour_tz "tt timestamp"; then + q "CREATE FOREIGN TABLE hw (id bigint, ts timestamptz) + SERVER ice OPTIONS (metadata_path '$MD_th')" >/dev/null + check_text "hour() on a naive source read as timestamptz in New York matches iceberg_scan" \ + "$(qny "SELECT string_agg(id::text, ',' ORDER BY id) FROM hw WHERE ts >= TIMESTAMPTZ '2021-03-01 06:00:00+00'")" \ + "$(qny "SELECT string_agg(id::text, ',' ORDER BY id) + FROM pgcolumnar.iceberg_scan('$MD_th') AS t(id bigint, ts timestamptz) + WHERE ts >= TIMESTAMPTZ '2021-03-01 06:00:00+00'")" + check_text "hour() on a timestamptz source read as timestamp in New York matches iceberg_scan" \ + "$(qny "SELECT string_agg(id::text, ',' ORDER BY id) FROM hn WHERE tt <= TIMESTAMP '2021-02-28 20:00:00'")" \ + "$(qny "SELECT string_agg(id::text, ',' ORDER BY id) + FROM pgcolumnar.iceberg_scan('$MD_hn') AS t(id bigint, tt timestamp) + WHERE tt <= TIMESTAMP '2021-02-28 20:00:00'")" +fi + # ---- an invalid option is rejected by the validator ------------------------- check "an unknown table option is refused (FDW_INVALID_OPTION_NAME)" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA 2>&1 </dev/null + pgc_teardown +} +trap open_uploads_teardown EXIT INT TERM + +pg_restart_env() { + pgc_pg "pg_ctl -D '$PGC_PGDATA' stop -m fast -w" >/dev/null 2>&1 + pgc_pg "$* pg_ctl -D '$PGC_PGDATA' -l '$PGC_LOGFILE' start -w" >/dev/null 2>&1 + for _ in $(seq 1 30); do [ -n "$(q 'SELECT 1')" ] && return 0; sleep 0.5; done + echo "FATAL: cluster did not come back"; exit 1 +} +# Lines of the server log matching $1: a number, or nothing when it can't be read. +logcount() { + local _n + _n="$(pgc_pg "grep -c '$1' '$PGC_LOGFILE'" 2>/dev/null)" + case "$_n" in '' | *[!0-9]*) ;; *) echo "$_n" ;; esac +} +survived() { # survived BEFORE -> "1" when the server answers and no backend died since + local _up _now + _up="$(q 'SELECT 1')"; _now="$(logcount 'terminated by signal')" + if ! [[ "$1" =~ ^[0-9]+$ && "$_now" =~ ^[0-9]+$ ]]; then echo "no (crash count unreadable)" + elif [ "$_up" = 1 ] && [ "$_now" = "$1" ]; then echo 1 + else echo "no ([$_up], $(( _now - $1 )) backend crashes)"; fi +} +# The export's own open uploads under prefix $1 (part-0000..0003): a number, or +# nothing when the server's record is missing. +own_open() { + [ -f "$OPEN" ] || return 0 + grep -cE "^$BUCKET/$1/part-000[0-3]\.parquet\$" "$OPEN" | tr -d ' ' +} +# Every other open upload under prefix $1, sorted, space-separated. +foreign_open() { + [ -f "$OPEN" ] || { echo "no record"; return 0; } + grep -E "^$BUCKET/$1/" "$OPEN" | grep -vE "^$BUCKET/$1/part-000[0-3]\.parquet\$" | + sort | tr '\n' ' ' | sed 's/ $//' +} +part_puts() { grep -cE "^PUT /$BUCKET/$1/part-[0-9]+\.parquet .*partNumber=" "$S3_LOG" | tr -d ' '; } +# Wait until all 4 of the export's uploads under $1 are open; then report what +# the store holds: "4 open, 0 parts" means every worker waits for its upload id. +wait_unanswered() { + for _ in $(seq 1 300); do + [ "$(own_open "$1")" = 4 ] && break + sleep 0.1 + done + echo "$(own_open "$1") open, $(part_puts "$1") parts" +} +# Wait until all 4 of the export's uploads under $1 are open and 4 parts were +# sent; "4 open, sending parts" means every worker holds its upload id. +wait_sending() { + for _ in $(seq 1 300); do + [ "$(own_open "$1")" = 4 ] && [ "$(part_puts "$1")" -ge 4 ] && break + sleep 0.1 + done + if [ "$(part_puts "$1")" -ge 4 ]; then + echo "$(own_open "$1") open, sending parts" + else + echo "$(own_open "$1") open, $(part_puts "$1") parts" + fi +} +cancel_export() { + q "SELECT pg_cancel_backend(pid) FROM pg_stat_activity + WHERE query LIKE '%parallel_export_parquet%' AND backend_type = 'client backend' + AND pid <> pg_backend_pid()" >/dev/null +} +terminate_workers() { + q "SELECT count(pg_terminate_backend(pid)) FROM pg_stat_activity + WHERE backend_type = 'pgcolumnar parallel_export worker'" >/dev/null +} +# start_export PREFIX OUTFILE: a 4-worker export to s3://bucket/PREFIX in the +# background; its pid is left in $bgpid. +start_export() { + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -v VERBOSITY=verbose \ + -c "SELECT pgcolumnar.parallel_export_parquet('ou_big'::regclass, 's3://$BUCKET/$1', 4)" \ + >"$2" 2>&1 & + bgpid=$! +} +completed_parts() { ls "$PGC_WORKDIR/$BUCKET/$1/" 2>/dev/null | grep -c 'part-.*\.parquet$'; } + +mkdir -p "$PGC_WORKDIR/$BUCKET" +python3 "$(dirname "${BASH_SOURCE[0]}")/objstore_http_server.py" \ + --dir "$PGC_WORKDIR" --port "$S3_PORT" --log "$S3_LOG" \ + --sigv4-key "$AKID" --sigv4-secret "$SECRET" --sigv4-region "$REGION" \ + --list-page-size 2 \ + --seed-upload "$BUCKET/op/part-0004.parquet" --seed-upload "$BUCKET/op/notes.parquet" \ + --seed-upload "$BUCKET/oc/part-0004.parquet" --seed-upload "$BUCKET/oc/notes.parquet" \ + --seed-upload "$BUCKET/ot/part-0004.parquet" --seed-upload "$BUCKET/ot/notes.parquet" \ + --seed-upload "$BUCKET/od/part-0004.parquet" --seed-upload "$BUCKET/od/notes.parquet" \ + > "$PGC_WORKDIR/ou_server.out" 2>&1 & +SRV_PID=$! +for _ in $(seq 1 50); do + grep -q READY "$PGC_WORKDIR/ou_server.out" 2>/dev/null && break + sleep 0.1 +done +check "premise: sigv4 fixture server is up" \ + "$(grep -c READY "$PGC_WORKDIR/ou_server.out" 2>/dev/null)" "1" +check "premise: the foreign uploads were open before the exports" \ + "$(foreign_open op) $(foreign_open oc) $(foreign_open ot)" \ + "$BUCKET/op/notes.parquet $BUCKET/op/part-0004.parquet $BUCKET/oc/notes.parquet $BUCKET/oc/part-0004.parquet $BUCKET/ot/notes.parquet $BUCKET/ot/part-0004.parquet" + +pg_restart_env "AWS_ENDPOINT_URL='http://127.0.0.1:$S3_PORT'" \ + "AWS_ACCESS_KEY_ID='$AKID'" "AWS_SECRET_ACCESS_KEY='$SECRET'" \ + "AWS_REGION='$REGION'" + +# Enough rows that each of 4 workers writes more than one 256 KiB part, so each +# one starts a multipart upload. +psql_run "CREATE TABLE ou_big (id int8, pad text) USING pgcolumnar; + SELECT pgcolumnar.set_options('ou_big', stripe_row_limit => 4000); + INSERT INTO ou_big SELECT g, md5(g::text)||md5((g+1)::text) + FROM generate_series(1,400000) g;" >/dev/null +crashes0="$(logcount 'terminated by signal')" + +# ---- parts: the dispatcher is cancelled while every worker sends parts ---- +: > "$S3_LOG" +echo 20 > "$PGC_WORKDIR/__part_delay__" +start_export op "$PGC_WORKDIR/op.out" +check "premise: parts: every worker was sending parts when the cancel landed" \ + "$(wait_sending op)" "4 open, sending parts" +cancel_export +wait "$bgpid" 2>/dev/null || true +rm -f "$PGC_WORKDIR/__part_delay__" +check "parts: the export reports its own cancel (57014)" \ + "$(grep -m1 '^ERROR:' "$PGC_WORKDIR/op.out")" \ + "ERROR: 57014: canceling statement due to user request" +check_num "parts: no upload of this export is left open" "$(own_open op)" "0" +check_text "parts: the foreign uploads under the prefix are left open" \ + "$(foreign_open op)" "$BUCKET/op/notes.parquet $BUCKET/op/part-0004.parquet" +check "parts: the server survived" "$(survived "$crashes0")" "1" + +# ---- cancel: the dispatcher is cancelled while every POST ?uploads is held ---- +: > "$S3_LOG" +echo 20 > "$PGC_WORKDIR/__uploads_delay__" +start_export oc "$PGC_WORKDIR/oc.out" +check "premise: cancel: every worker's CreateMultipartUpload was unanswered when the cancel landed" \ + "$(wait_unanswered oc)" "4 open, 0 parts" +cancel_export +wait "$bgpid" 2>/dev/null || true +rm -f "$PGC_WORKDIR/__uploads_delay__" +check "cancel: the export reports its own cancel (57014)" \ + "$(grep -m1 '^ERROR:' "$PGC_WORKDIR/oc.out")" \ + "ERROR: 57014: canceling statement due to user request" +check_num "cancel: no upload of this export is left open" "$(own_open oc)" "0" +check_text "cancel: the foreign uploads under the prefix are left open" \ + "$(foreign_open oc)" "$BUCKET/oc/notes.parquet $BUCKET/oc/part-0004.parquet" +check_num "cancel: no completed object remains at the prefix" "$(completed_parts oc)" "0" +check "cancel: the server survived" "$(survived "$crashes0")" "1" + +# ---- terminate: the workers are terminated while every POST ?uploads is held ---- +: > "$S3_LOG" +echo 20 > "$PGC_WORKDIR/__uploads_delay__" +start_export ot "$PGC_WORKDIR/ot.out" +check "premise: terminate: every worker's CreateMultipartUpload was unanswered when the workers were terminated" \ + "$(wait_unanswered ot)" "4 open, 0 parts" +terminate_workers +wait "$bgpid" 2>/dev/null || true +rm -f "$PGC_WORKDIR/__uploads_delay__" +check "terminate: the export reports that a worker failed" \ + "$(grep -m1 '^ERROR:' "$PGC_WORKDIR/ot.out" | sed -E 's/worker [0-9]+ failed/worker N failed/')" \ + "ERROR: XX000: pgcolumnar.parallel_export_parquet worker N failed: worker exited without reporting a result" +check_num "terminate: no upload of this export is left open" "$(own_open ot)" "0" +check_text "terminate: the foreign uploads under the prefix are left open" \ + "$(foreign_open ot)" "$BUCKET/ot/notes.parquet $BUCKET/ot/part-0004.parquet" +check "terminate: the server survived" "$(survived "$crashes0")" "1" + +# ---- double cancel: a second cancel lands in the first cancel's part DELETE ---- +: > "$S3_LOG" +echo 20 > "$PGC_WORKDIR/__uploads_delay__" +echo 3 > "$PGC_WORKDIR/__delete_delay__" +errstart0="$(logcount 'errstart was not called')" +start_export od "$PGC_WORKDIR/od.out" +check "premise: double cancel: every worker's CreateMultipartUpload was unanswered when the first cancel landed" \ + "$(wait_unanswered od)" "4 open, 0 parts" +cancel_export +# The cleanup's first part DELETE (no query: not an upload abort) is logged +# on arrival and answered 3 s later. +deleting=no +for _ in $(seq 1 100); do + grep -qE "^DELETE /$BUCKET/od/part-[0-9]+\.parquet - " "$S3_LOG" && { deleting=yes; break; } + sleep 0.1 +done +check "premise: double cancel: the first cancel's cleanup was deleting a part when the second cancel landed" \ + "$deleting" "yes" +cancel_export +# Later DELETEs, the upload aborts among them, are answered at once. +rm -f "$PGC_WORKDIR/__delete_delay__" +wait "$bgpid" 2>/dev/null || true +rm -f "$PGC_WORKDIR/__uploads_delay__" +check "double cancel: the export reports its own cancel (57014)" \ + "$(grep -m1 '^ERROR:' "$PGC_WORKDIR/od.out")" \ + "ERROR: 57014: canceling statement due to user request" +errstart1="$(logcount 'errstart was not called')" +check "double cancel: the server log has no 'errstart was not called'" \ + "$( [[ "$errstart0" =~ ^[0-9]+$ && "$errstart1" =~ ^[0-9]+$ ]] && echo $(( errstart1 - errstart0 )) || echo "unreadable")" "0" +check_num "double cancel: no upload of this export is left open" "$(own_open od)" "0" +check_text "double cancel: the foreign uploads under the prefix are left open" \ + "$(foreign_open od)" "$BUCKET/od/notes.parquet $BUCKET/od/part-0004.parquet" +check "double cancel: the server survived" "$(survived "$crashes0")" "1" + +# ---- silent listing: the store does not answer ListMultipartUploads ---- +: > "$S3_LOG" +echo 20 > "$PGC_WORKDIR/__uploads_delay__" +echo 120 > "$PGC_WORKDIR/__list_uploads_delay__" +start_export os "$PGC_WORKDIR/os.out" +check "premise: silent listing: every worker's CreateMultipartUpload was unanswered when the cancel landed" \ + "$(wait_unanswered os)" "4 open, 0 parts" +t0="$(date +%s)" +cancel_export +wait "$bgpid" 2>/dev/null || true +t1="$(date +%s)" +rm -f "$PGC_WORKDIR/__uploads_delay__" "$PGC_WORKDIR/__list_uploads_delay__" +check "silent listing: the cleanup asked the store for its open uploads" \ + "$(grep -cE "^GET /$BUCKET .*uploads=" "$S3_LOG" | tr -d ' ')" "1" +check "silent listing: the export reports its own cancel (57014)" \ + "$(grep -m1 '^ERROR:' "$PGC_WORKDIR/os.out")" \ + "ERROR: 57014: canceling statement due to user request" +check "silent listing: the export returned within 30 s of the cancel" \ + "$( (( t1 - t0 <= 30 )) && echo yes || echo "no, after $(( t1 - t0 )) s")" "yes" +check "silent listing: the server survived" "$(survived "$crashes0")" "1" + +pgc_summary diff --git a/test/objstore_http_server.py b/test/objstore_http_server.py index bf1ca45b9..c4fa6b87d 100755 --- a/test/objstore_http_server.py +++ b/test/objstore_http_server.py @@ -31,6 +31,25 @@ # to hold the dispatcher's cleanup DELETE in flight while a # second cancel lands (#1741). # +# Multipart introspection and hooks (#1743), read per request from --dir: +# __open_uploads__ written by the server, never read: the key of every +# upload created and neither completed nor aborted, one +# per line, rewritten on each change. These are the +# billable orphans a real store would keep. +# __uploads_delay__ a number of seconds: CreateMultipartUpload (POST +# ?uploads) creates the upload at once and answers only +# after that long, so a suite can end the client while +# the upload exists and its id has not reached it. +# __part_delay__ a number of seconds: every part PUT is stored at once +# and answered only after that long, so a suite can end +# the client while its uploads are in progress. +# __list_uploads_delay__ a number of seconds: ListMultipartUploads (GET +# ?uploads) is answered only after that long, a silent +# store for the client's bounded wait. +# --seed-upload KEY (repeatable) opens an upload at KEY ("bucket/key") before +# the first request: an upload some other writer left, which a client must not +# abort as its own. +# # SigV4 verification mode (#393 M2): with --sigv4-key/--sigv4-secret/ # --sigv4-region, every request must carry a valid AWS Signature Version 4 # Authorization header. The signature is recomputed here with python's stdlib @@ -312,6 +331,54 @@ def esc(s): self.end_headers() self.wfile.write(body) + def _do_list_uploads(self): + # ListMultipartUploads (#1743): the uploads in progress whose key starts + # with `prefix`, sorted by key then upload id, paged by --list-page-size + # with key-marker/upload-id-marker as S3 does it. + _hook_sleep(self.server.rootdir, "__list_uploads_delay__") + q = self._qdict() + bucket = self.path.partition("?")[0].lstrip("/").split("/", 1)[0] + prefix = q.get("prefix", "") + kmark = q.get("key-marker") + umark = q.get("upload-id-marker", "") + base = os.path.join(self.server.rootdir, bucket) + with self.server.uploads_lock: + ups = [] + for uid, up in self.server.uploads.items(): + key = os.path.relpath(up["local"], base).replace(os.sep, "/") + if key.startswith(prefix) and not key.startswith(".."): + ups.append((key, uid)) + ups.sort() + if kmark is not None: + ups = [u for u in ups if u > (kmark, umark)] + page_size = self.server.list_page_size + page = ups[:page_size] + truncated = len(ups) > page_size + + def esc(x): + return (x.replace("&", "&").replace("<", "<") + .replace(">", ">")) + parts = ["", + "", + "%s" % esc(bucket), + "%s" % esc(prefix), + "%d" % page_size, + "%s" % ("true" if truncated else "false")] + if truncated and page: + parts.append("%s" % esc(page[-1][0])) + parts.append("%s" % esc(page[-1][1])) + for key, uid in page: + parts.append("%s%s" + "2026-10-22T00:00:00.000Z" + % (esc(key), esc(uid))) + parts.append("") + body = "".join(parts).encode() + self.send_response(200) + self.send_header("Content-Type", "application/xml") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + def do_GET(self): self._log_line() if not self._sigv4_check(): @@ -320,6 +387,9 @@ def do_GET(self): if "list-type=2" in _query: self._do_list_v2(_query) return + if "uploads" in self._qdict(): + self._do_list_uploads() + return mode, local = self._resolve() if mode == "hangup": self.close_connection = True @@ -378,15 +448,6 @@ def do_GET(self): # ---- write side (#394): plain PUT, and a faithful multipart emulation -- - def _open_uploads(self): - # /__open_uploads__ lists the key of every upload created and - # neither completed nor aborted, one per line, rewritten on each change: - # the billable orphans a real store would keep. Call with uploads_lock. - root = self.server.rootdir - with open(os.path.join(root, "__open_uploads__"), "w") as f: - for up in self.server.uploads.values(): - f.write(os.path.relpath(up["local"], root) + "\n") - def _qdict(self): q = {} for part in self.path.partition("?")[2].split("&"): @@ -418,6 +479,7 @@ def do_PUT(self): self._deny(code, "InternalError") return up["parts"][int(q["partNumber"])] = self._body + _hook_sleep(self.server.rootdir, "__part_delay__") etag = hashlib.md5(self._body).hexdigest() self.send_response(200) self.send_header("ETag", '"%s"' % etag) @@ -447,7 +509,8 @@ def do_POST(self): self.server.upload_seq += 1 uid = "up/%d+x" % self.server.upload_seq self.server.uploads[uid] = {"local": local, "parts": {}} - self._open_uploads() + _write_open_uploads(self.server) + _hook_sleep(self.server.rootdir, "__uploads_delay__") body = ("" "%s" "" % uid).encode() @@ -479,7 +542,7 @@ def do_POST(self): self._deny(400, "InvalidPart") return del self.server.uploads[q["uploadId"]] - self._open_uploads() + _write_open_uploads(self.server) os.makedirs(os.path.dirname(local), exist_ok=True) with open(local, "wb") as f: for n, _ in listed: @@ -507,7 +570,7 @@ def do_DELETE(self): with self.server.uploads_lock: up = self.server.uploads.pop(q["uploadId"], None) if up is not None: - self._open_uploads() + _write_open_uploads(self.server) if up is None: self._deny(404, "NoSuchUpload") return @@ -518,6 +581,22 @@ def do_DELETE(self): self.end_headers() +def _write_open_uploads(srv): + # /__open_uploads__: one key per open upload. Call with uploads_lock. + with open(os.path.join(srv.rootdir, "__open_uploads__"), "w") as f: + for up in srv.uploads.values(): + f.write(os.path.relpath(up["local"], srv.rootdir) + "\n") + + +def _hook_sleep(rootdir, name): + # Sleep for the number of seconds in /, when it exists. + try: + with open(os.path.join(rootdir, name)) as fh: + time.sleep(float(fh.read().strip() or 0)) + except (OSError, ValueError): + pass + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--dir", required=True) @@ -535,6 +614,8 @@ def main(): "bucket (S3 virtual-host addressing, #621)") ap.add_argument("--list-page-size", type=int, default=1000, help="ListObjectsV2 keys per page; small values force paging") + ap.add_argument("--seed-upload", action="append", default=[], + help="open a multipart upload at bucket/key before serving") args = ap.parse_args() if args.tls_cert: @@ -554,6 +635,11 @@ def main(): srv.uploads = {} srv.uploads_lock = threading.Lock() srv.upload_seq = 0 + for key in args.seed_upload: + srv.upload_seq += 1 + srv.uploads["seed/%d+x" % srv.upload_seq] = { + "local": os.path.join(args.dir, key), "parts": {}} + _write_open_uploads(srv) open(args.log, "a").close() # Readiness marker for the suite: print once the socket is bound. print("READY", flush=True) diff --git a/test/parquet_export.sh b/test/parquet_export.sh index 5619714d2..116cbb482 100755 --- a/test/parquet_export.sh +++ b/test/parquet_export.sh @@ -7,7 +7,8 @@ # empty/unicode text, bytea) that mirrors a heap table, reads it back, and # asserts every value equals the heap oracle. pyarrow provides exact typed # values for the comparison; if the DuckDB CLI is present it is used as a second -# independent reader (row count and a sum). Also covers multiple row groups, an +# independent reader (row counts, a non-null count and a decimal), and without it +# those four checks are skipped by name. Also covers multiple row groups, an # empty table, and error cases. # # Requires pyarrow (python3-pyarrow); if it is not importable the suite skips @@ -27,13 +28,32 @@ if ! python3 -c 'import pyarrow.parquet' 2>/dev/null; then pgc_skip pyarrow "pyarrow not available; skipping Parquet export verification" fi -expect_error() { - local label="$1" sql="$2" - if psql_run "$sql" >/dev/null 2>&1; then - check "$label (expected error)" "succeeded" "error" +# expect_refusal NAME SQL MESSAGE -- SQL must fail with MESSAGE in its error. Any +# error used to count, so a backend crash or an unrelated failure passed as the +# refusal (#1341): with the columnar check removed, exporting a heap failed for +# another reason, and with the type check removed the backend aborted, and both +# passed. psql's exit status says whether it failed; the message says why. +expect_refusal() { + local name="$1" sql="$2" msg="$3" out got + if out="$(psql_run "$sql" 2>&1)"; then + got="succeeded" + elif grep -qF -- "$msg" <<<"$out"; then + got="refused: $msg" else - check "$label" "error" "error" + got="other error: $(head -n1 <<<"$out")" fi + check "$name" "$got" "refused: $msg" +} + +# duckdb_skip NAME -- without the duckdb CLI, record the DuckDB check NAME as +# skipped, by name. The four checks used to be left out without a word, so a run +# without DuckDB counted four checks fewer and said nothing (#1341). The twin +# declines the same four by the same names. A skip, not check_unrunnable: the +# suites job installs no DuckDB CLI, and an unrunnable check makes the suite +# INCOMPLETE, which fails its major there. CI holds the four in the twin instead, +# where PGC_REQUIRE names duckdb and turns a decline into a failure. +duckdb_skip() { + check_skip "$1" "SKIP $1 (the duckdb CLI is not installed)" "duckdb CLI not installed" } PARQUET="$PGC_WORKDIR/t.parquet" @@ -85,7 +105,13 @@ def feq(x,y): if math.isnan(x) and math.isnan(y): return True return x==y return x==y -ok = set(ar)==set(pg) and all(all(feq(a,b) for a,b in zip(ar[k],pg[k])) for k in pg) +# The ids the generator wrote, so that two empty readings cannot agree (#1341). +ids=set(range(1,80001))|{100001,100002,100003} +# And the file's own row count: rows are compared by id, so a row written twice +# would otherwise collapse into one. +if set(pg)!=ids or set(ar)!=ids or tbl.num_rows!=len(ids): + print("ROWS heap %d parquet %d, not the generator's 80003"%(len(pg),tbl.num_rows)); sys.exit(0) +ok = all(all(feq(a,b) for a,b in zip(ar[k],pg[k])) for k in pg) print("MATCH" if ok else "MISMATCH") PY )" @@ -104,8 +130,13 @@ if command -v duckdb >/dev/null 2>&1; then dcount="$(duckdb -noheader -list -c "SELECT count(*) FROM read_parquet('$PARQUET');" 2>/dev/null | tr -d '[:space:]')" check "duckdb row count" "$dcount" "80003" dsum_d="$(duckdb -noheader -list -c "SELECT count(b) FROM read_parquet('$PARQUET');" 2>/dev/null | tr -d '[:space:]')" - pcnt="$(q "SELECT count(b) FROM t_col;")" - check "duckdb non-null count(b)" "$dsum_d" "$pcnt" + # The generator's count, 80000 rows plus the two edge rows with a b: t_col is + # the table under test, so its own count agreed with whatever it lost, and with + # an empty read on both sides (#1341). + check_num "duckdb non-null count(b)" "$dsum_d" "80002" +else + duckdb_skip "duckdb row count" + duckdb_skip "duckdb non-null count(b)" fi # Empty table. @@ -122,10 +153,12 @@ check "empty parquet readable" "$er" "0 a,b" # Errors. echo "-- argument validation" -expect_error "reject non-columnar table" "SELECT pgcolumnar.export_parquet('t_heap', '$PGC_WORKDIR/x.parquet');" +expect_refusal "reject non-columnar table" "SELECT pgcolumnar.export_parquet('t_heap', '$PGC_WORKDIR/x.parquet');" \ + "is not a columnar table" psql_run "CREATE TABLE t_un (a int, p point) USING pgcolumnar;" psql_run "INSERT INTO t_un VALUES (1, '(1,2)');" -expect_error "reject unsupported type" "SELECT pgcolumnar.export_parquet('t_un', '$PGC_WORKDIR/x.parquet');" +expect_refusal "reject unsupported type" "SELECT pgcolumnar.export_parquet('t_un', '$PGC_WORKDIR/x.parquet');" \ + 'column "p" has type point, which columnar parquet export does not support' # --------------------------------------------------------------------------- # Extended type coverage: date, time, timestamp(+tz), uuid, numeric(p,s) as @@ -195,6 +228,9 @@ pg = {} for r in csv.reader(open(sys.argv[2],newline='')): pg[int(r[0])] = (int(r[0]), I(r[1]), I(r[2]), I(r[3]), I(r[4]), N(r[5]), N(r[6]), N(r[7]), N(r[8]), N(r[9])) +# The ids the INSERT wrote, so that two empty readings cannot agree (#1341). +if set(pg)!={1,2,3,4} or set(ar)!={1,2,3,4} or t.num_rows!=4: + print("ROWS heap %r parquet %r, not ids 1..4"%(sorted(pg),ids)); sys.exit(0) print("MATCH" if ar==pg else "MISMATCH %r %r"%(ar,pg)) PY )" @@ -216,6 +252,9 @@ if command -v duckdb >/dev/null 2>&1; then check "duckdb reads decimal" "$dnum" "12345.6789" dext="$(duckdb -noheader -list -c "SELECT count(*) FROM read_parquet('$EXT');" 2>/dev/null | tr -d '[:space:]')" check "duckdb extended row count" "$dext" "4" +else + duckdb_skip "duckdb reads decimal" + duckdb_skip "duckdb extended row count" fi pgc_summary diff --git a/test/parquet_export_stats.sh b/test/parquet_export_stats.sh index e2e00ea68..0fa69141c 100755 --- a/test/parquet_export_stats.sh +++ b/test/parquet_export_stats.sh @@ -612,7 +612,11 @@ check_num "a date predicate skips" \ "$(skipped_for "WHERE d < DATE '2000-01-05'")" "3" check_num "a timestamp predicate skips" \ "$(skipped_for "WHERE ts < TIMESTAMP '2000-01-01 00:30:00'")" "3" -# Documented refusals, reachable from our own exported files for the first time. +# CROSS-TYPE CONSTANTS. These arms measured a documented refusal: a constant whose +# type was not the column's skipped nothing. Since #1712 the reader compares through +# the operator family's cross-type function, so each now skips exactly as its +# same-type twin beside it does, and the soundness arms below cover every one. +# The comments that follow describe how each pair was built. # # Each runs beside a predicate that differs in ONE respect and does skip. Without # that pair the checks are worthless: every group's max(i8) exceeds 5 and every @@ -621,8 +625,8 @@ check_num "a timestamp predicate skips" \ # not evidence of a refusal. check_num "control: the same column skips when the constant matches its type" \ "$(skipped_for "WHERE i8 < 5::bigint")" "$NGROUPS" -check_num "a cross-type constant skips nothing, on the identical predicate" \ - "$(skipped_for "WHERE i8 < 5::int")" "0" +check_num "a cross-type constant skips as the matching one does, on the identical predicate" \ + "$(skipped_for "WHERE i8 < 5::int")" "$NGROUPS" check_num "control: a float < predicate below every bound skips" \ "$(skipped_for "WHERE f4 < 0.4::real")" "$NGROUPS" # The reader refuses > and >= on a float column outright, because Parquet @@ -644,8 +648,8 @@ check_num "a float >= predicate above every bound still skips nothing" \ # constant cannot produce, which is what the pair is for. check_num "a quoted literal takes a bigint column's type and skips" \ "$(skipped_for "WHERE i8 < '5'")" "$NGROUPS" -check_num "the same digits unquoted are an integer, and skip nothing" \ - "$(skipped_for "WHERE i8 < 5")" "0" +check_num "the same digits unquoted are an integer, and skip as well" \ + "$(skipped_for "WHERE i8 < 5")" "$NGROUPS" check_num "digits too wide for an integer are a bigint, and skip" \ "$(skipped_for "WHERE i8 < 5000000000")" "3" # A decimal-point literal is a `numeric`, and PostgreSQL resolves @@ -665,12 +669,12 @@ check_num "control: the quoted form casts the constant to the column's type" \ "$(filter_has "WHERE i8 < '5'" '::bigint')" "1" check_num "a quoted literal takes a real column's type and skips" \ "$(skipped_for "WHERE f4 < '0.4'")" "$NGROUPS" -check_num "the same literal unquoted is double precision, and skips nothing" \ - "$(skipped_for "WHERE f4 < 0.4")" "0" +check_num "the same literal unquoted is double precision, and skips as well" \ + "$(skipped_for "WHERE f4 < 0.4")" "$NGROUPS" check_num "a quoted literal takes a smallint column's type and skips" \ "$(skipped_for "WHERE i2 < '500'")" "1" -check_num "the same digits unquoted are an integer, and skip nothing (int2)" \ - "$(skipped_for "WHERE i2 < 500")" "0" +check_num "the same digits unquoted are an integer, and skip as well (int2)" \ + "$(skipped_for "WHERE i2 < 500")" "1" # ADORNED is not the same as quoted. A literal that names its own type is that # type, not `unknown`, so it does not take the column's and the exact-type @@ -682,16 +686,12 @@ check_num "an adorned literal of the column's own type skips" \ "$(skipped_for "WHERE d < DATE '2000-01-05'")" "3" check_num "the same date unadorned is unknown, takes the column's type and skips" \ "$(skipped_for "WHERE d < '2000-01-05'")" "3" -check_num "the same date adorned as a timestamp does not skip" \ - "$(skipped_for "WHERE d < TIMESTAMP '2000-01-05 00:00:00'")" "0" -# That last arm is a documented-behaviour guard rather than evidence, and the -# distinction is worth stating rather than leaving for a reader to discover. -# No mutation in this suite's ledger reddens it: with the reader's -# exact-constant-type refusal deleted, the timestamp datum read as a date lands -# outside this fixture's range in the direction that prunes nothing, so the arm -# still reports 0 for a second reason. What IS falsifiable is the typing the -# sentence rests on, and the planner will say it: an adorned literal keeps its -# own type beside a bare Var, while the unadorned form resolves to the column's. +check_num "the same date adorned as a timestamp skips as well" \ + "$(skipped_for "WHERE d < TIMESTAMP '2000-01-05 00:00:00'")" "3" +# The typing the pair rests on is falsifiable, and the planner will say it: an +# adorned literal keeps its own type beside a bare Var, while the unadorned form +# resolves to the column's. Since #1712 both forms skip, through different +# comparison functions. check_num "an adorned literal keeps its own type, not the column's" \ "$(filter_has "WHERE d < TIMESTAMP '2000-01-05 00:00:00'" '::timestamp')" "1" check_num "control: the unadorned form resolves to the column's type instead" \ @@ -704,8 +704,8 @@ check_num "control: the unadorned form resolves to the column's type instead" \ # and the other prunes none. check_num "control: a digit string that fits an integer prunes every group" \ "$(skipped_for "WHERE id < -1")" "$NGROUPS" -check_num "digits too wide for an integer are a bigint, so an int4 column refuses" \ - "$(skipped_for "WHERE id < -5000000000")" "0" +check_num "digits too wide for an integer are a bigint, and an int4 column skips as well" \ + "$(skipped_for "WHERE id < -5000000000")" "$NGROUPS" # Soundness. A skipped group emits nothing, and the executor's recheck cannot # recover a row that never arrived, so every predicate above must return exactly @@ -755,6 +755,12 @@ fdw_vs_oracle "the FDW result equals the oracle for [id < -5000000000]" \ "id < -5000000000" fdw_vs_oracle "the FDW result equals the oracle for [alln IS NULL]" \ "alln IS NULL" +fdw_vs_oracle "the FDW result equals the oracle for [i8 < 5::int]" \ + "i8 < 5::int" +fdw_vs_oracle "the FDW result equals the oracle for [i8 < 5]" \ + "i8 < 5" +fdw_vs_oracle "the FDW result equals the oracle for [f4 < 0.4]" \ + "f4 < 0.4" # ---- the other direction: the instrument can report a bound that IS there --- # diff --git a/test/parquet_skip_cross_type.sh b/test/parquet_skip_cross_type.sh new file mode 100755 index 000000000..e3e4d119a --- /dev/null +++ b/test/parquet_skip_cross_type.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# +# Parquet row-group skipping uses a constant of another type in the column's +# operator family (#1712). +# +# The foreign table's group-exclusion check refused any constant whose type was +# not the column's. A bare integer literal is int4, so the most ordinary filter +# on a bigint column, `WHERE id > 150000`, read every row group; only +# `150000::int8` skipped. The same held for a float4 constant on a float8 column. +# The native columnar path already compares through the operator family's +# cross-type comparison function, and this suite asks the same of Parquet. +# +# A file of 200,000 ascending rows in 4 row groups of 50,000. Each arm reads the +# skip count from EXPLAIN (ANALYZE) and, beside it, the answer, compared with the +# same query written with a constant of the column's own type. +# +# Usage: test/parquet_skip_cross_type.sh [PG_CONFIG] + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +if ! python3 -c 'import pyarrow.parquet' 2>/dev/null; then + pgc_skip pyarrow "pyarrow not available; Parquet cross-type skip suite needs it" +fi + +W="$PGC_WORKDIR" +NGROUPS="$(python3 - "$W" <<'PY' +import sys, pyarrow as pa, pyarrow.parquet as pq +w = sys.argv[1] +ids = list(range(200000)) +pq.write_table(pa.table({"id": pa.array(ids, pa.int64()), + "f": pa.array([float(i) for i in ids], pa.float64())}), + f"{w}/xt.parquet", row_group_size=50000) +# NaN sits in the second group; Parquet statistics leave it out of min and max +pq.write_table(pa.table({"f": pa.array([1.0, 2.0, 3.0, float("nan")], pa.float64())}), + f"{w}/xn.parquet", row_group_size=2) +# a naive timestamp column in 2 row groups: group 0 spans New York's 2024-03-10 +# spring-forward gap (ids 0-2), group 1 is the hour before it (ids 3-5) +import datetime as dt +gap = [dt.datetime(2024, 3, 10, h, m) for h, m in ((2, 0), (2, 30), (3, 0), (1, 0), (1, 10), (1, 20))] +pq.write_table(pa.table({"id": pa.array(range(6), pa.int64()), "ts": pa.array(gap, pa.timestamp("us"))}), + f"{w}/xg.parquet", row_group_size=3) +print(pq.ParquetFile(f"{w}/xt.parquet").metadata.num_row_groups, + pq.ParquetFile(f"{w}/xg.parquet").metadata.num_row_groups) +PY +)" +GGROUPS="${NGROUPS#* }" +NGROUPS="${NGROUPS%% *}" +chmod 644 "$W/xt.parquet" "$W/xn.parquet" "$W/xg.parquet" +psql_run "CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; + CREATE FOREIGN TABLE xt (id int8, f float8) SERVER pq OPTIONS (path '$W/xt.parquet'); + CREATE FOREIGN TABLE xn (f float8) SERVER pq OPTIONS (path '$W/xn.parquet');" + +# skipped QUERY -> the "Row Groups Skipped" count EXPLAIN (ANALYZE) reports, or the error +skipped() { + local out + out="$(q "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) $1" 2>&1)" + grep -oE 'Row Groups Skipped: [0-9]+' <<<"$out" | grep -oE '[0-9]+$' || echo "no reading: $(head -1 <<<"$out")" +} + +check_num "premise: the file holds 4 row groups (pyarrow's own count)" "$NGROUPS" "4" +check_num "control: an int8 constant on the int8 column skips 3 of the 4 row groups" \ + "$(skipped "SELECT count(*) FROM xt WHERE id > 150000::int8")" "3" +check_num "an int4 constant on the int8 column skips 3 of the 4 row groups" \ + "$(skipped "SELECT count(*) FROM xt WHERE id > 150000")" "3" +check_num "and returns the rows the int8 constant returns" \ + "$(q "SELECT count(*) FROM xt WHERE id > 150000;")" "$(q "SELECT count(*) FROM xt WHERE id > 150000::int8;")" +check_num "an int4 constant in an equality on the int8 column skips 3 of the 4 row groups" \ + "$(skipped "SELECT count(*) FROM xt WHERE id = 150001")" "3" +check_num "a constant on the left of the operator is compared the same way" \ + "$(skipped "SELECT count(*) FROM xt WHERE 150000 < id")" "3" +# `<` because the float rule never skips on `>` or `>=` (a NaN row satisfies them +# and sits outside min/max), and a NEGATIVE constant because a float4 misread as +# float8 is a tiny positive number: then group 0 (min 0.0) would not be skipped. +check_num "a float4 constant below every row skips all 4 row groups of the float8 column" \ + "$(skipped "SELECT count(*) FROM xt WHERE f < -1::float4")" "4" +check_num "and returns the rows the float8 constant returns" \ + "$(q "SELECT count(*) FROM xt WHERE f < 50000::float4;")" "$(q "SELECT count(*) FROM xt WHERE f < 50000::float8;")" + +# A NaN constant is read by its own type. Read as the column's float8, a float4 NaN +# is some other number, and the group holding the NaN row would be skipped. +check_num "a float4 NaN constant still finds the float8 column's NaN row" \ + "$(q "SELECT count(*) FROM xn WHERE f = 'NaN'::float4;")" "1" + +# A timestamp column and a timestamptz constant (OffgridwithJD's review of #1737). +# The cross-type comparison converts each statistic, a wall time, to an instant in +# the session zone, and that conversion is not monotonic across a spring-forward +# gap. In New York, group 0's max 03:00 becomes 07:00Z, yet its row 02:30 becomes +# 07:30Z. Every session below runs in New York; the heap copy is the oracle. +psql_run "CREATE FOREIGN TABLE xg (id int8, ts timestamp) SERVER pq OPTIONS (path '$W/xg.parquet'); + CREATE TABLE xg_heap AS SELECT * FROM xg;" +qny() { PGOPTIONS="-c TimeZone=America/New_York" q "$1"; } +ids() { qny "SELECT coalesce(string_agg(id::text, ',' ORDER BY id), 'none') FROM $1 WHERE $2;"; } +check_num "premise: the gap file holds 2 row groups (pyarrow's own count)" "$GGROUPS" "2" +check_text "premise: in New York the heap copy finds row 1 for ts > 07:15Z, ts >= 07:15Z and ts = 07:30Z" \ + "$(ids xg_heap "ts > '2024-03-10 07:15+00'::timestamptz") $(ids xg_heap "ts >= '2024-03-10 07:15+00'::timestamptz") $(ids xg_heap "ts = '2024-03-10 07:30+00'::timestamptz")" \ + "1 1 1" +check_text "a timestamptz constant after a spring-forward gap finds the timestamp column's row the heap finds (>)" \ + "$(ids xg "ts > '2024-03-10 07:15+00'::timestamptz")" "$(ids xg_heap "ts > '2024-03-10 07:15+00'::timestamptz")" +check_text "a timestamptz constant after a spring-forward gap finds the timestamp column's row the heap finds (>=)" \ + "$(ids xg "ts >= '2024-03-10 07:15+00'::timestamptz")" "$(ids xg_heap "ts >= '2024-03-10 07:15+00'::timestamptz")" +check_text "a timestamptz constant equal to a row's instant after a spring-forward gap finds that row, as the heap does" \ + "$(ids xg "ts = '2024-03-10 07:30+00'::timestamptz")" "$(ids xg_heap "ts = '2024-03-10 07:30+00'::timestamptz")" +check_num "control: a timestamp constant on the timestamp column skips 1 of the 2 row groups" \ + "$(PGOPTIONS="-c TimeZone=America/New_York" skipped "SELECT count(*) FROM xg WHERE ts > '2024-03-10 02:00'::timestamp")" "1" +check_text "and returns the rows the heap returns" \ + "$(ids xg "ts > '2024-03-10 02:00'::timestamp")" "$(ids xg_heap "ts > '2024-03-10 02:00'::timestamp")" + +pgc_summary diff --git a/test/parquet_timestamp_zone.sh b/test/parquet_timestamp_zone.sh new file mode 100755 index 000000000..86ed5105f --- /dev/null +++ b/test/parquet_timestamp_zone.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# +# A Parquet timestamp read into the other PostgreSQL timestamp type follows +# PostgreSQL's cast semantics (#1706 item 2). +# +# Parquet marks each timestamp column isAdjustedToUTC: true for an instant, +# false for a wall time with no zone. The reader ignored the flag and returned +# the raw microseconds for either target type, so two of the four cases +# disagreed with PostgreSQL, in a session whose zone is not UTC: +# +# file read as before now, as PostgreSQL casts +# naive timestamptz taken as UTC the wall time in the session zone +# adjusted timestamp the UTC wall the session-zone wall time +# +# The same-kind reads (naive into timestamp, adjusted into timestamptz) are +# unchanged, and are the controls. +# +# A WALL TIME IS NOT MONOTONIC IN ITS INSTANT across a spring-forward gap: +# PostgreSQL reads the nonexistent 02:30 as standard time, so in New York on +# 2024-03-10 the wall times 02:00, 02:30, 03:00 become 07:00Z, 07:30Z, 07:00Z. +# A row group's converted min and max then need not bound its converted rows, +# so row-group skipping must not use them. The last arm reads such a group +# through the foreign table with a qual only the 02:30 row meets. +# +# The other arms: the naive file read again in a second zone (Tokyo), so a reader +# that hard-codes a zone fails; the repeated hour of New York's 2024-11-03 +# fall-back, which must resolve as PostgreSQL's cast resolves it; and a file +# whose timestamp carries only the legacy TIMESTAMP_MICROS converted type, no +# logical type. The Parquet spec defines that type as UTC-adjusted. The fixture +# test/fixtures/parquet/legacy_timestamp_micros.parquet is real: pgcolumnar's own +# export_parquet before #1707 (main at 0c5e924d) wrote it, from one timestamp +# column holding 2024-06-01 12:00, so the file holds the instant 12:00Z. +# +# Usage: test/parquet_timestamp_zone.sh [PG_CONFIG] + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +if ! python3 -c 'import pyarrow.parquet' 2>/dev/null; then + pgc_skip pyarrow "pyarrow not available; Parquet timestamp zone suite needs it" +fi + +W="$PGC_WORKDIR" +# Every session of this suite runs in New York, so each reading below is in that zone. +export PGOPTIONS="-c TimeZone=America/New_York" +psql_run "CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet;" + +python3 - "$W" <<'PY' +import sys, datetime as dt, pyarrow as pa, pyarrow.parquet as pq +w = sys.argv[1] +noon = dt.datetime(2024, 6, 1, 12, 0) +# naive: timestamp[us], isAdjustedToUTC=false; adjusted: tz=UTC, isAdjustedToUTC=true +pq.write_table(pa.table({"t": pa.array([noon], pa.timestamp("us"))}), f"{w}/naive.parquet") +pq.write_table(pa.table({"t": pa.array([noon], pa.timestamp("us", tz="UTC"))}), f"{w}/adjusted.parquet") +# one row group spanning New York's 2024-03-10 spring-forward gap +gap = [dt.datetime(2024, 3, 10, 2, 0), dt.datetime(2024, 3, 10, 2, 30), dt.datetime(2024, 3, 10, 3, 0)] +pq.write_table(pa.table({"t": pa.array(gap, pa.timestamp("us"))}), f"{w}/gap.parquet", row_group_size=3) +# the repeated hour of New York's 2024-11-03 fall-back +fb = [dt.datetime(2024, 11, 3, 1, 30)] +pq.write_table(pa.table({"t": pa.array(fb, pa.timestamp("us"))}), f"{w}/fallback.parquet") +PY +cp "$(dirname "${BASH_SOURCE[0]}")/fixtures/parquet/legacy_timestamp_micros.parquet" "$W/legacy.parquet" +for f in naive adjusted gap fallback legacy; do chmod 644 "$W/$f.parquet"; done + +check "premise: pyarrow wrote the naive file without isAdjustedToUTC and the other with it" \ + "$(python3 -c " +import sys, pyarrow.parquet as pq +a = lambda f: pq.ParquetFile(f).schema_arrow.field('t').type.tz +print(a('$W/naive.parquet'), a('$W/adjusted.parquet'))")" "None UTC" +# pyarrow derives a logical type from a converted type and says so in this flag +check "premise: the legacy fixture carries the TIMESTAMP_MICROS converted type and no logical type" \ + "$(python3 -c " +import pyarrow.parquet as pq +c = pq.ParquetFile('$W/legacy.parquet').metadata.schema.column(0) +print(c.converted_type, 'is_from_converted_type=true' in str(c.logical_type))")" "TIMESTAMP_MICROS True" + +psql_run "CREATE TABLE n_ts (t timestamp) USING pgcolumnar; CREATE TABLE n_tz (t timestamptz) USING pgcolumnar; + CREATE TABLE a_ts (t timestamp) USING pgcolumnar; CREATE TABLE a_tz (t timestamptz) USING pgcolumnar;" +for t in n_ts n_tz; do psql_run "SELECT pgcolumnar.import_parquet('$t', '$W/naive.parquet');"; done +for t in a_ts a_tz; do psql_run "SELECT pgcolumnar.import_parquet('$t', '$W/adjusted.parquet');"; done + +check_text "a naive file read into timestamptz is its wall time in the session zone" \ + "$(q "SELECT t::text FROM n_tz;")" "2024-06-01 12:00:00-04" +check_text "and that is what PostgreSQL's cast from timestamp gives" \ + "$(q "SELECT (SELECT t FROM n_tz) = '2024-06-01 12:00'::timestamp::timestamptz;")" "t" +check_text "an adjusted file read into timestamp is the session-zone wall time of its instant" \ + "$(q "SELECT t::text FROM a_ts;")" "2024-06-01 08:00:00" +check_text "and that is what PostgreSQL's cast from timestamptz gives" \ + "$(q "SELECT (SELECT t FROM a_ts) = '2024-06-01 12:00+00'::timestamptz::timestamp;")" "t" +check_text "control: a naive file read into timestamp keeps its wall time" \ + "$(q "SELECT t::text FROM n_ts;")" "2024-06-01 12:00:00" +check_text "control: an adjusted file read into timestamptz keeps its instant" \ + "$(q "SELECT t::text FROM a_tz;")" "2024-06-01 08:00:00-04" + +# The round trip through pgcolumnar's own export, in the same zone. This holds only +# because the exporter writes timestamp as naive (#1707): with the reader honouring +# the flag and the old exporter marking timestamp adjusted, the timestamp came back +# 08:00, four hours early. +psql_run "CREATE TABLE rt_src (ts timestamp, tz timestamptz) USING pgcolumnar; + INSERT INTO rt_src VALUES ('2024-06-01 12:00', '2024-06-01 12:00'); + CREATE TABLE rt_dst (ts timestamp, tz timestamptz) USING pgcolumnar;" +psql_run "SELECT pgcolumnar.export_parquet('rt_src', '$W/rt.parquet');" +psql_run "SELECT pgcolumnar.import_parquet('rt_dst', '$W/rt.parquet');" +check_text "an export and import round trip keeps timestamp and timestamptz in a non-UTC session" \ + "$(q "SELECT ts::text || ' | ' || tz::text FROM rt_dst;")" "2024-06-01 12:00:00 | 2024-06-01 12:00:00-04" + +psql_run "CREATE FOREIGN TABLE f_gap (t timestamptz) SERVER pq OPTIONS (path '$W/gap.parquet');" +check_text "the gap file's wall times 02:00, 02:30 and 03:00 become, sorted, the instants 07:00Z, 07:00Z and 07:30Z" \ + "$(q "SELECT string_agg(to_char(t AT TIME ZONE 'UTC', 'HH24:MI'), ',' ORDER BY t) FROM f_gap;")" \ + "07:00,07:00,07:30" +check_text "a row group spanning a spring-forward gap is not skipped on its converted min and max" \ + "$(q "SELECT count(*) FROM f_gap WHERE t > '2024-03-10 07:15+00';")" "1" + +# A second zone: the same naive file read through a foreign table in Tokyo. +psql_run "CREATE FOREIGN TABLE f_naive (t timestamptz) SERVER pq OPTIONS (path '$W/naive.parquet');" +qtk() { PGOPTIONS="-c TimeZone=Asia/Tokyo" q "$1"; } +check_text "a naive file read into timestamptz in Tokyo is its wall time there, as the cast gives" \ + "$(qtk "SELECT t::text || ' ' || (t = '2024-06-01 12:00'::timestamp::timestamptz)::text FROM f_naive;")" \ + "2024-06-01 12:00:00+09 true" +check_text "and the Tokyo reading is a different instant from the New York one" \ + "$(qtk "SELECT ((SELECT t FROM f_naive) <> (SELECT t FROM n_tz))::text;")" "true" + +# The repeated hour: 01:30 happens twice in New York on 2024-11-03. +psql_run "CREATE FOREIGN TABLE f_fb (t timestamptz) SERVER pq OPTIONS (path '$W/fallback.parquet');" +check_text "a naive wall time in the fall-back hour read into timestamptz is what the cast gives" \ + "$(q "SELECT t::text || ' ' || (t = '2024-11-03 01:30'::timestamp::timestamptz)::text FROM f_fb;")" \ + "2024-11-03 01:30:00-05 true" + +# The legacy converted type, read as the instant the spec defines it to be. +psql_run "CREATE FOREIGN TABLE f_leg_ts (t timestamp) SERVER pq OPTIONS (path '$W/legacy.parquet'); + CREATE FOREIGN TABLE f_leg_tz (t timestamptz) SERVER pq OPTIONS (path '$W/legacy.parquet');" +check_text "a legacy TIMESTAMP_MICROS file read into timestamp is the session-zone wall time of its instant" \ + "$(q "SELECT t::text || ' ' || (t = '2024-06-01 12:00+00'::timestamptz::timestamp)::text FROM f_leg_ts;")" \ + "2024-06-01 08:00:00 true" +check_text "a legacy TIMESTAMP_MICROS file read into timestamptz keeps its instant" \ + "$(q "SELECT t::text FROM f_leg_tz;")" "2024-06-01 08:00:00-04" + +pgc_summary diff --git a/test/planner_choice_quality.sh b/test/planner_choice_quality.sh index 7e8ccd1a7..2c1342f81 100755 --- a/test/planner_choice_quality.sh +++ b/test/planner_choice_quality.sh @@ -91,10 +91,18 @@ check "premise: the index key is correlated, which is the case that misprices" \ FROM pg_stats WHERE tablename='pq' AND attname='k'")" "yes" # Time a query under a given setting, and report which scan node ran. -run_plan() { # run_plan -> " " - local node ms out +# +# A FAILED RUN IS NOT A FAST ONE. psql prints "Time:" after a statement that +# errored as well as after one that succeeded, so this used to read a plan that +# raised an error in 2 ms as a plan that answered in 2 ms, and the ratio below +# passed with the chosen plan unable to run at all (measured: a custom scan that +# raises at execution left the suite PASSED). psql's exit status under +# ON_ERROR_STOP is the signal now, and the statement-timeout cancel is told +# apart by its own message rather than by any line containing "timeout". +run_plan() { # run_plan -> "\t" + local node ms out rc node=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -Atq -c "$1" -c "EXPLAIN (COSTS OFF) $2" 2>&1 | + -d "$PGC_DB" -v ON_ERROR_STOP=1 -Atq -c "$1" -c "EXPLAIN (COSTS OFF) $2" 2>&1 | grep -oE 'Index Only Scan|Index Scan|Bitmap Heap Scan|Custom Scan \([A-Za-z]+\)|Seq Scan' | head -1) # No execution under PGC_SKIP_TIMING. The plan name above comes from EXPLAIN, # so every premise still holds; only the wall clock is unavailable. @@ -107,11 +115,18 @@ run_plan() { # run_plan -> " " # warm against a 3x bound -- the ordering alone could push a healthy ratio # over it. The discarded run also populates shared buffers for the timed one. env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -Atq -c "SET statement_timeout='${PLAN_TIMEOUT}s';" -c "$1" -c "$2" >/dev/null 2>&1 + -d "$PGC_DB" -v ON_ERROR_STOP=1 -Atq -c "SET statement_timeout='${PLAN_TIMEOUT}s';" -c "$1" -c "$2" >/dev/null 2>&1 out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -Atq -c "SET statement_timeout='${PLAN_TIMEOUT}s';" -c "$1" -c '\timing on' -c "$2" 2>&1) - if grep -qiE 'timeout|canceling' <<<"$out"; then - printf '%s\tTIMEOUT\n' "${node:-unknown}"; return + -d "$PGC_DB" -v ON_ERROR_STOP=1 -Atq -c "SET statement_timeout='${PLAN_TIMEOUT}s';" -c "$1" -c '\timing on' -c "$2" 2>&1) + rc=$? + if [ "$rc" != 0 ]; then + if grep -q 'canceling statement due to statement timeout' <<<"$out"; then + printf '%s\tTIMEOUT\n' "${node:-unknown}" + else + out=$(grep -m1 -E 'ERROR|FATAL|error' <<<"$out" | tr '\t' ' ') + printf '%s\tERROR (%s)\n' "${node:-unknown}" "${out:-psql exit $rc}" + fi + return fi ms=$(grep -oE 'Time: [0-9.]+ ms' <<<"$out" | tail -1 | grep -oE '[0-9.]+') # Tab-delimited, because a node name contains spaces: "Custom Scan @@ -122,9 +137,15 @@ run_plan() { # run_plan -> " " } # The planner's choice against the best alternative it declined. +# +# THE CHECK NAMES CARRY THE LABEL AND NOTHING ELSE. They used to carry the bound, +# the timeout and the node that won the comparison, so the same check had one +# name when it passed ("... alternative (Index Scan)") and another when it failed +# ("... alternative (Custom Scan (PgColumnarScan))") -- a red that no green run +# had ever named. Those values are in the display line now, not the name. choice_vs_best() { # choice_vs_best