Piddiplatsch

Documentation · Overview slides

Piddiplatsch processes ESGF STAC publication records from Kafka and registers persistent identifiers (PIDs) with the Handle System. It supports CMIP6, CMIP6Plus, CMIP7, and CORDEX-CMIP6, with project-specific routing and mapping.

Curious by nature. Persistent by design.

Quick start

Install with Conda and the development tools:

git clone https://github.com/ESGF/piddiplatsch.git
cd piddiplatsch
conda env create
conda activate piddi
make develop

Create your local configuration and replace the connection and credential placeholders before running:

cp etc/esgf-example.toml custom.toml
# Edit custom.toml for your site; keep credentials out of Git.
piddi config validate
piddi --help

Start harvesting and mapping Kafka messages into prepared Handle JSONL files:

piddi consume

By default, consume saves raw messages and prepares Handles without publishing to a Handle service. Kafka access is required; see Configuration for site settings.

Run in stages

You can also harvest a small sample, then map it separately:

piddi harvest --limit 100
piddi map --project cmip6 --date last

Once your Handle service profile is configured, publish a completed daily file:

piddi publish --project cmip6 --date yesterday

# Or publish an inclusive range of available daily files
piddi publish --project cmip6 --from 2026-10-08 --to today-1

Use piddi COMMAND --help for options. The Operations guide explains date selection, publication, retries, logging, and monitoring.

Learn more

• Architecture and project plugins
• Recovery and retry
• Production deployment with Ansible
• Contributing: development, testing, and local Docker services. Run make test for unit and integration tests.