dbscrape

dbscrape writes a local snapshot of PostgreSQL table and column metadata under /tmp/tables.

Install

Requires Bash 3.2 or newer, curl, and the PostgreSQL psql client.

curl -fsSL https://raw.githubusercontent.com/frittlechasm/dbscrape/main/install.sh | bash

The installer places dbscrape in $HOME/.local/bin, which must be on PATH. Upgrade an installed copy to the latest release with:

dbscrape update

To choose another directory:

curl -fsSL https://raw.githubusercontent.com/frittlechasm/dbscrape/main/install.sh \
  | bash -s -- --bin-dir "$HOME/bin"

To uninstall, remove the installed executable:

rm "$HOME/.local/bin/dbscrape"

Usage

dbscrape <username> <host[:port]> <database>
dbscrape update

dbscrape prompts for the password without echoing it. For automation, provide the password through the runtime's secret store:

PGPASSWORD="$DB_PASSWORD" dbscrape app_user localhost app_database

Bracketed IPv6 hosts are supported, for example [::1]:5433.

Output

A successful scrape replaces /tmp/tables with a fresh snapshot:

/tmp/tables/
├── tables.txt
├── table-paths.jsonl
├── users/
│   ├── columns.txt
│   └── full-details.txt
└── audit.events/
    ├── columns.txt
    └── full-details.txt

• tables.txt lists schema-qualified table names.
• table-paths.jsonl maps output directories to exact PostgreSQL identifiers.
• columns.txt lists columns in their defined order.
• full-details.txt contains the output of psql's \d command.

Supported tables and names

dbscrape includes ordinary, non-partition tables from user schemas. It excludes views, materialized views, foreign tables, partitioned tables, partitions, information_schema, and PostgreSQL-managed schemas.

Folder names are lowercase and portable:

• Order Items becomes order-items.
• path/table becomes path-table.
• 100% done becomes 100%-done.
• café becomes cafe.

Tables in public use <table>. Other schemas use <schema>.<table>. If normalized names collide, each directory receives a stable hash suffix. Use table-paths.jsonl to recover the exact schema and table names.

Run behavior

• Scrapes up to five tables concurrently.
• Uses one discovery connection and one connection per table for both output files.
• Keeps the previous snapshot if discovery or scraping fails.
• Refuses to replace /tmp/tables when it is a symbolic link or is not owned by the current user.

Development

Run the local behavior suite:

./tests/run.sh

The suite uses a fake psql to check CLI behavior, file handling, failures, and concurrency. Changes to SQL or psql formatting also need verification against a real PostgreSQL database; the fake supplies precomputed table records.