A command-line tool that moves data between databases, files and object storage, and transforms or anonymizes it on the way. One self-contained binary; nothing is staged in between.
dotnet tool install -g dtpipe # or download a binary from the releases page
dtpipe --helpLinux, macOS and Windows, x64 and arm64. The .NET runtime, DuckDB and every database driver are inside the binary. Details: docs/install.md.
Anonymize a table on the way out, so the clear values never reach the disk:
dtpipe \
-i "pg:Host=localhost;Database=prod;Username=postgres" \
--query "SELECT * FROM users" \
--fake "email:internet.email" \
--fake "name:name.fullName" \
--mask "phone:###-****" \
--null "ssn" \
-o anonymized_users.parquetA first pipeline in six steps, with no database to set up: docs/quickstart.md.
| Family | Prefix |
|---|---|
| Databases | pg:, mysql:, sqlite:, duck:, mssql:, ora: |
| Files | csv:, jsonl:, parquet:, arrow:, xml:, or just the file extension |
| Object storage | s3://bucket/key.parquet, azure://container/blob.csv |
| Test endpoints | generate:N (a source), null: and checksum: (sinks) |
dtpipe providers lists what your binary carries.
| docs/ | Start here: install, quickstart, concepts, a guide per task, a page per connection |
| REFERENCE.md | The flags in common use, the YAML job schema, provider options, the MCP tools. dtpipe --help lists every flag |
| EXTENDING.md | Writing a new adapter or transformer |
Beyond the command line: one pipeline across several hosts
(docs/distributed.md), dtpipe studio to design and preview jobs in a
local page, with an AI assistant if you give it a model, and dtpipe mcp for an assistant you
already use.
MIT