Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ Any catalog implementing the [Iceberg REST Catalog spec](https://iceberg.apache.
```sh
cd example/single
docker compose up -d --wait
docker compose --profile workload up -d workload
```

Open `http://localhost:8123/play` and run:
Expand All @@ -228,7 +229,9 @@ Open `http://localhost:8123/play` and run:
SELECT * FROM rideshare.`rideshare.rides`
```

You should see new rows appearing as the simulator drives PG.
You should see new rows appearing as the workload drives PG.

`./run.sh` in the same directory runs it as a smoke test: the mixed workload — deletes, large and rolled-back transactions, key changes, TOAST, schema changes — with a crash, external compactions and snapshot expiry alongside, then a row-by-row comparison of Postgres with ClickHouse. See [`example/single`](example/single).

## Configuration

Expand Down
2 changes: 1 addition & 1 deletion example/blue-green/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ services:

iceberg-rest:
# Reuse the same REST catalog image as the single example.
build: ../../../single/iceberg-rest
build: ../single/iceberg-rest
environment:
AWS_ACCESS_KEY_ID: admin
AWS_SECRET_ACCESS_KEY: password
Expand Down
1 change: 1 addition & 0 deletions example/single/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
out/
55 changes: 55 additions & 0 deletions example/single/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Single-node example and smoke test

A rideshare app's Postgres → pg2iceberg → Iceberg REST catalog + MinIO, with
ClickHouse to query the Iceberg tables and a mixed workload to drive
Postgres.

Bring it up and watch rows arrive:

```sh
cd example/single
docker compose up -d --wait
docker compose --profile workload up -d workload # DURATION=300 seconds of writes
```

Then query `http://localhost:8123/play`, e.g.
``SELECT * FROM rideshare.`rideshare.rides` ``.

Or run it as a smoke test, which ends with a row-by-row comparison of
Postgres and ClickHouse and exits non-zero if they differ (logs in `out/`):

```sh
./run.sh # DURATION=300 by default
docker compose --profile workload down -v
```

## What runs

Two concurrent writers ([`workload.py`](workload.py)) mix:

- `simulate.py`'s rideshare actions: multi-table transactions of inserts and
updates;
- deletes, and transactions that insert, update and delete the same rows;
- rolled-back transactions and primary-key changes;
- TOAST: ~12 KB `drivers.notes` values, and updates that leave them alone.

`drivers` has Postgres's default replica identity (a delete carries only the
key) and is partitioned by `status`, so a status change moves a row between
partitions ([`setup.sql`](setup.sql), [`config.yaml`](config.yaml)).

On a timeline, the workload also runs a 5000-row insert, a bulk delete, a
`TRUNCATE`, adding / dropping / re-adding a column, a bulk update and a status
flip of every driver; and [`run.sh`](run.sh) runs `pg2iceberg compact` twice,
SIGKILLs and restarts pg2iceberg, and runs `pg2iceberg maintain` with a 1m
snapshot retention.

## What's checked

- [`compare.py`](compare.py): every row and column of each table, Postgres
against ClickHouse (an independent Iceberg reader), plus ClickHouse's
`count()`.
- `pg2iceberg verify`, in `out/verify.log`.
- The replication slot catches up once the workload stops (to within a few
hundred bytes: the coordinator's own state lives in this Postgres).

On Colima, bind mounts only work under `$HOME`: run from a checkout there.
61 changes: 61 additions & 0 deletions example/single/compare.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""Compare Postgres with ClickHouse reading the Iceberg tables pg2iceberg
wrote: every row, every column rendered the same way on both sides, and
ClickHouse's `count()`. Exits non-zero on any difference."""

import os
import subprocess
import sys
import urllib.request

HERE = os.path.dirname(os.path.abspath(__file__))
TABLES = ["riders", "drivers", "rides", "payments", "ratings"]


def pg(sql):
out = subprocess.run(
["docker", "compose", "exec", "-T", "postgres", "psql", "-U", "postgres", "-d", "rideshare",
"-At", "-F", "\t", "-c", sql],
cwd=HERE, capture_output=True, text=True, check=True)
return out.stdout.splitlines()


def ch(sql):
req = urllib.request.Request("http://localhost:8123", data=(sql + " FORMAT TSVRaw").encode())
return urllib.request.urlopen(req).read().decode().splitlines()


def render(name, ty):
"""The same text for a value on both sides."""
if name == "notes": # TOAST-sized: compare digests
return f"coalesce(md5({name}), 'NULL')", f"ifNull(lower(hex(MD5({name}))), 'NULL')"
if ty.startswith("timestamp"):
return (f"coalesce((extract(epoch FROM {name}) * 1000000)::bigint::text, 'NULL')",
f"ifNull(toString(toUnixTimestamp64Micro({name})), 'NULL')")
if ty == "numeric":
return f"coalesce(trim_scale({name})::text, 'NULL')", f"ifNull(toString({name}), 'NULL')"
return f"coalesce({name}::text, 'NULL')", f"ifNull(toString({name}), 'NULL')"


ok = True
for table in TABLES:
cols = [tuple(r.split("\t")) for r in pg(
"SELECT column_name, data_type FROM information_schema.columns "
f"WHERE table_schema = 'public' AND table_name = '{table}' ORDER BY ordinal_position")]
pg_exprs, ch_exprs = zip(*(render(n, t) for n, t in cols))
iceberg = f"rideshare.`rideshare.{table}`"
pg_rows = pg(f"SELECT {', '.join(pg_exprs)} FROM {table}")
ch_rows = ch(f"SELECT {', '.join(ch_exprs)} FROM {iceberg}")
ch_count = int(ch(f"SELECT count() FROM {iceberg}")[0])
only_pg, only_ch = set(pg_rows) - set(ch_rows), set(ch_rows) - set(pg_rows)
dupes = len(ch_rows) - len(set(ch_rows))
good = not (only_pg or only_ch or dupes) and ch_count == len(pg_rows)
ok &= good
print(f"{table:9} pg={len(pg_rows):6} iceberg={len(ch_rows):6} count()={ch_count:6} "
f"only_pg={len(only_pg)} only_iceberg={len(only_ch)} duplicates={dupes} "
f"{'OK' if good else 'DIFF'}")
for r in sorted(only_pg)[:3]:
print(" pg :", r[:200])
for r in sorted(only_ch)[:3]:
print(" iceberg :", r[:200])
print("ALL MATCH" if ok else "MISMATCHES FOUND")
sys.exit(0 if ok else 1)
7 changes: 4 additions & 3 deletions example/single/config.yaml
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
tables:
- name: public.riders
- name: public.drivers
iceberg:
# A non-key column under DEFAULT replica identity: deletes carry only
# the key, and a status change moves a row between partitions.
partition: ["status"]
- name: public.rides
- name: public.payments
- name: public.ratings

source:
mode: logical

postgres:
host: postgres
port: 5432
database: rideshare
user: postgres
password: postgres

logical:
publication_name: pg2iceberg_pub
slot_name: pg2iceberg_slot
Expand All @@ -29,6 +31,5 @@ sink:
s3_region: us-east-1
flush_interval: 10s
flush_rows: 1000
flush_bytes: 67108864

state: {}
113 changes: 29 additions & 84 deletions example/single/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,51 +1,44 @@
# Postgres -> pg2iceberg -> Iceberg REST catalog + MinIO, with ClickHouse to
# query the tables and a mixed workload to drive Postgres. See README.md.
name: pg2iceberg-single
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: rideshare
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- "5502:5432"
ports: ["5502:5432"]
volumes:
- ./schema.sql:/docker-entrypoint-initdb.d/01-schema.sql
- ./seed.sql:/docker-entrypoint-initdb.d/02-seed.sql
- pgdata:/var/lib/postgresql/data
- ./setup.sql:/docker-entrypoint-initdb.d/03-setup.sql
command: ["postgres", "-c", "wal_level=logical", "-c", "max_replication_slots=4", "-c", "max_wal_senders=4"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
test: ["CMD-SHELL", "pg_isready -U postgres -d rideshare"]
interval: 2s
timeout: 5s
retries: 10
retries: 30

minio:
# minio/minio + minio/mc are no longer on Docker Hub; pgsty/* is a
# community-maintained drop-in fork (same binaries and entrypoints).
image: pgsty/minio:RELEASE.2026-08-04T00-00-00Z
command: server /data --console-address ":9001"
command: server /data
environment:
MINIO_ROOT_USER: admin
MINIO_ROOT_PASSWORD: password
ports:
- "9000:9000"
- "9001:9001"
ports: ["9000:9000"]
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 2s
timeout: 5s
retries: 5
retries: 15

create-bucket:
image: pgsty/mc:RELEASE.2026-09-16T00-00-00Z
depends_on:
minio:
condition: service_healthy
minio: {condition: service_healthy}
entrypoint: >
/bin/sh -c "
mc alias set myminio http://minio:9000 admin password;
mc mb --ignore-existing myminio/warehouse;
exit 0;
"
/bin/sh -c "mc alias set m http://minio:9000 admin password && mc mb --ignore-existing m/warehouse"

iceberg-postgres:
image: postgres:16-alpine
Expand All @@ -57,7 +50,7 @@ services:
test: ["CMD-SHELL", "pg_isready -U iceberg"]
interval: 2s
timeout: 5s
retries: 10
retries: 15

iceberg-rest:
build: ./iceberg-rest
Expand All @@ -73,50 +66,32 @@ services:
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
CATALOG_S3_ENDPOINT: http://minio:9000
CATALOG_S3_PATH__STYLE__ACCESS: "true"
ports:
- "8181:8181"
ports: ["8181:8181"]
depends_on:
iceberg-postgres:
condition: service_healthy
create-bucket:
condition: service_completed_successfully
iceberg-postgres: {condition: service_healthy}
create-bucket: {condition: service_completed_successfully}
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:8181/v1/config || exit 1"]
interval: 3s
timeout: 5s
retries: 10
retries: 30

pg2iceberg:
build:
context: ../..
dockerfile: Dockerfile
# Dockerfile's ENTRYPOINT is the binary; pass the subcommand here.
# Config is sourced from environment variables (see below); no
# `--config` file needed for this example.
command: ["run", "--config", "/etc/pg2iceberg/config.yaml"]
environment:
RUST_LOG: info
volumes:
- ./config.yaml:/etc/pg2iceberg/config.yaml:ro
environment:
POSTGRES_URL: postgresql://postgres:postgres@postgres:5432/rideshare
TABLES: public.riders,public.drivers,public.rides,public.payments,public.ratings
ICEBERG_CATALOG_URL: http://iceberg-rest:8181
WAREHOUSE: s3://warehouse/
NAMESPACE: rideshare
S3_ENDPOINT: http://minio:9000
S3_ACCESS_KEY: admin
S3_SECRET_KEY: password
ports:
- "9091:9090"
depends_on:
postgres:
condition: service_healthy
iceberg-rest:
condition: service_healthy
postgres: {condition: service_healthy}
iceberg-rest: {condition: service_healthy}

clickhouse:
image: clickhouse/clickhouse-server:26.2
ports:
- "8123:8123"
image: clickhouse/clickhouse-server:26.8
ports: ["8123:8123"]
environment:
AWS_ACCESS_KEY_ID: admin
AWS_SECRET_ACCESS_KEY: password
Expand All @@ -125,46 +100,16 @@ services:
- ./clickhouse/users.xml:/etc/clickhouse-server/users.d/iceberg.xml
- ./clickhouse/init.sql:/docker-entrypoint-initdb.d/init.sql
depends_on:
iceberg-rest:
condition: service_healthy

prometheus:
image: prom/prometheus:v3.4.1
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
depends_on:
pg2iceberg:
condition: service_started
iceberg-rest: {condition: service_healthy}

grafana:
image: grafana/grafana:11.6.0
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
ports:
- "3001:3000"
depends_on:
- prometheus

simulate:
workload:
image: python:3.12-slim
depends_on:
postgres:
condition: service_healthy
profiles: ["workload"]
environment:
DATABASE_URL: postgres://postgres:postgres@postgres:5432/rideshare
DURATION: ${DURATION:-300}
volumes:
- ./simulate.py:/app/simulate.py
- ../../scripts/requirements.txt:/app/requirements.txt
- ./workload.py:/app/workload.py
working_dir: /app
command: sh -c "pip install -q -r requirements.txt && python simulate.py --interval 5"

volumes:
pgdata:
command: sh -c "pip install -q psycopg2-binary && python -u workload.py"
1 change: 0 additions & 1 deletion example/single/grafana/dashboards/pg2iceberg.json

This file was deleted.

11 changes: 0 additions & 11 deletions example/single/grafana/provisioning/dashboards/default.yml

This file was deleted.

This file was deleted.

Loading
Loading