Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,19 @@ true until the next version shipped.
over a failed attempt's file is allowed.

### Changed
- **A Parquet timestamp read into the other PostgreSQL timestamp type is
converted in the session's time zone.** The reader ignored the file's
`isAdjustedToUTC` flag. A wall-time column read into `timestamptz` was taken as
UTC (12:00 read as 08:00-04 in a New York session), and a UTC-adjusted column
read into `timestamp` kept its UTC wall time. Both now follow PostgreSQL's casts.
Same-type reads are unchanged, and so is any read in a UTC session (#1706).
A legacy `TIMESTAMP_MILLIS` or `TIMESTAMP_MICROS` column with no logical type,
as pgcolumnar's own exporter wrote before #1707, is read as UTC-adjusted, as
the Parquet format defines it. The Iceberg foreign table no longer prunes on a
`year()`, `month()`, `day()` or `hour()` partition whose source column is the
other timestamp type: its partition values are in the source's type, so they
no longer bound the converted rows.
`test/parquet_timestamp_zone.sh`, `test/iceberg_fdw.sh` and their twins.

- `GROUP BY` is vectorized by default (#1292).
`pgcolumnar.enable_group_vectorization` is now `on`. The planner takes the
Expand Down Expand Up @@ -431,6 +444,15 @@ true until the next version shipped.
ended the cleanup early: the segments stayed pinned until the server restarted, and
the second cancel was reported instead of the first. The cleanup now finishes
whatever arrives, in the time a single cancel already took (#1745).
- **Parquet row-group skipping now uses a constant of another integer or float
type.** A bare integer literal is `int4`, so `WHERE id > 150000` on a `bigint`
column of a Parquet foreign table read every row group; only `150000::int8`
skipped. The check now compares through the operator family's cross-type
comparison, as the native columnar scan already did. Answers were always correct;
this only reads less (#1712). A `timestamp` column compared with a `timestamptz`
constant still does not skip: across a spring-forward gap the session zone's
conversion does not keep the statistics' order. `test/parquet_skip_cross_type.sh`
and `test_parquet_skip_cross_type.py`.
- **`export_parquet` no longer marks a `timestamp` column as a UTC instant.** Both
`timestamp` and `timestamptz` were written as the converted type `TIMESTAMP_MICROS`,
which the Parquet spec defines as adjusted to UTC. Every other reader (pyarrow,
Expand Down
11 changes: 8 additions & 3 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -1091,9 +1091,14 @@ correct rows; these are the conditions under which it can skip at all:
bytea, uuid, and boolean columns are filtered but never skipped, whatever their
statistics. A `numeric` column follows its storage: one written as an INT32 or
INT64 DECIMAL does skip, one written as a byte-array DECIMAL does not.
- The constant's type must match the column's type exactly. A cross-type
comparison such as `ts >= DATE '2026-01-01'` against a `timestamp` column, or
`bigint_col > 5::int`, does not skip.
- The constant may be of another type in the operator's family. So
`bigint_col > 5` skips, though `5` is an `int4`. So do `real_col < 0.4` and
`ts >= DATE '2026-01-01'` on a `timestamp` column. A comparison that casts the
column instead does not skip. An example is `bigint_col < 5.0`, where `5.0` is a
`numeric`. A `timestamp` column compared with a `timestamptz` constant does not
skip either. The comparison turns each wall time into an instant in the session's
time zone. Across a spring-forward gap that does not keep their order, so the
statistics would not bound the rows.
- The row group's statistics must carry both a minimum and a maximum, and the
interval must not be inverted. Two cases decode to an interval that the reader
does not trust for skipping. The first is an unsigned Parquet column that
Expand Down
18 changes: 18 additions & 0 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,24 @@ EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events_parquet WHERE ts >= '2026-01-
-- Files: 4
```

**Timestamps and time zones.** A Parquet timestamp column holds either instants
(UTC-adjusted) or wall times with no zone. Read into the matching type
(`timestamptz` for instants, `timestamp` for wall times), a value comes back
unchanged. Read into the other type, it is converted as PostgreSQL's own cast
converts it, in the session's `TimeZone`:

- a wall time read into `timestamptz` is taken as local time in that zone, like
`'2024-06-01 12:00'::timestamp::timestamptz`;
- an instant read into `timestamp` becomes the local wall time, like
`'2024-06-01 12:00+00'::timestamptz::timestamp`.

A foreign-table scan does not skip row groups of a column read this way, because
the conversion does not keep order across a daylight-saving change. For the same
reason the Iceberg foreign table does not prune files on a `year()`, `month()`,
`day()` or `hour()` partition of such a column. A column marked only with the
older `TIMESTAMP_MILLIS` or `TIMESTAMP_MICROS` type, with no `isAdjustedToUTC`
flag, holds instants: the Parquet format defines it so.

The `path` may also be an object-storage URL. Every read and export function, and
both foreign-data wrappers, accept an `s3://`, `http://`, or `https://` URL where
they accept a local path.
Expand Down
54 changes: 53 additions & 1 deletion src/columnar_iceberg.c
Original file line number Diff line number Diff line change
Expand Up @@ -1116,6 +1116,50 @@ ice_field_id_by_name(JsonbContainer *fieldsc, uint32 nf, const char *nm)
return -1;
}

/*
* Whether the current-schema field named `nm` is a timestamp of the OTHER kind
* than the PostgreSQL column `typid` (timestamp vs timestamptz, any unit). The
* reader converts such a column's rows as PostgreSQL's casts do, in the session
* zone (#1706), but the writer computed its temporal partition cells in the
* source's own kind, so those cells do not bound the converted rows.
*/
static bool
ice_field_timestamp_kind_differs(JsonbContainer *fieldsc, uint32 nf,
const char *nm, Oid typid)
{
uint32 j;

if (typid != TIMESTAMPOID && typid != TIMESTAMPTZOID)
return false;
for (j = 0; j < nf; j++)
{
JsonbValue *sf = getIthJsonbValueFromContainer(fieldsc, j);
JsonbValue *v;
bool src_tz;

if (sf == NULL || sf->type != jbvBinary)
continue;
v = ice_field(sf->val.binary.data, "name");
if (v == NULL || v->type != jbvString ||
(int) strlen(nm) != v->val.string.len ||
strncmp(nm, v->val.string.val, v->val.string.len) != 0)
continue;
v = ice_field(sf->val.binary.data, "type");
if (v == NULL || v->type != jbvString)
return false;
if (v->val.string.len >= 11 &&
strncmp(v->val.string.val, "timestamptz", 11) == 0)
src_tz = true;
else if (v->val.string.len >= 9 &&
strncmp(v->val.string.val, "timestamp", 9) == 0)
src_tz = false;
else
return false;
return src_tz != (typid == TIMESTAMPTZOID);
}
return false;
}

/*
* Map the current partition spec's IDENTITY fields to foreign-table columns, for
* FDW pruning. On return *out_attno[k] is the 1-based attno of the table column
Expand Down Expand Up @@ -1581,7 +1625,10 @@ PgColumnarIcebergDayMap(const char *path, const PgColumnarObjStoreConfig *cfg,
* default spec's partition-cell positions and 1-based source attnos for the
* fields carrying that transform. The FDW routes each mapped field by its source
* column type: the coarse temporal compiler handles the timestamp cases, the
* exact day() compiler handles a day() on a date.
* exact day() compiler handles a day() on a date. A field whose source is a
* timestamp of the other kind than its column (timestamp vs timestamptz) is left
* out, so nothing prunes on it: its cells are in the source's kind, its rows are
* converted in the session zone (#1706).
*/
void
PgColumnarIcebergTemporalMap(const char *path, const PgColumnarObjStoreConfig *cfg,
Expand Down Expand Up @@ -1676,6 +1723,11 @@ PgColumnarIcebergTemporalMap(const char *path, const PgColumnarObjStoreConfig *c
continue;
if (ice_field_id_by_name(fieldsc, nf, NameStr(att->attname)) == srcid)
{
/* the cells are in the source's kind, the rows in the column's */
if (ice_field_timestamp_kind_differs(fieldsc, nf,
NameStr(att->attname),
att->atttypid))
break;
pos[cnt] = (int) k;
attno[cnt] = a + 1;
cnt++;
Expand Down
Loading
Loading