skim

Stream a MySQL/MariaDB SQL dump and convert INSERT rows to JSON, JSONL, CSV, YAML, TOML, or Parquet — without loading the whole file into memory.

Features

• Streaming parser — handles arbitrarily large dump files
• Six output formats: JSON, JSONL, CSV, YAML, TOML, Parquet
• Filter to one or more tables with -t
• Format auto-detected from output file extension
• Optional progress bar (byte-progress for files, spinner for stdin)
• Parquet schema inferred from CREATE TABLE or from the first N rows

Installation

Linux / macOS

curl -fsSL https://raw.githubusercontent.com/aodihis/skim/master/scripts/install.sh | sh

Installs to ~/.local/bin/skim. Make sure that directory is in your PATH:

export PATH="$HOME/.local/bin:$PATH"

Windows (PowerShell)

irm https://raw.githubusercontent.com/aodihis/skim/master/scripts/install.ps1 | iex

Installs to %USERPROFILE%\.local\bin\skim.exe.

From source

cargo install --path .

Usage

skim [OPTIONS] [INPUT]

Argument	Description
INPUT	SQL dump file. Omit or use - to read from stdin.
-o, --output <FILE>	Output file. Omit or use - for stdout (default: stdout).
-f, --format <FORMAT>	Override output format (see formats below).
-t, --table <TABLE>	Only convert rows from this table. Repeatable.
--no-header	Suppress the CSV header row.
--no-progress	Disable the progress bar shown on stderr by default.
--dialect <DIALECT>	SQL dialect: mysql, postgres, or auto (default: auto).
--infer-rows <N>	Rows to buffer for Parquet schema inference when no CREATE TABLE is present (default: 1000).
--batch-size <N>	Rows per Arrow RecordBatch for Parquet output (default: 10000).
--max-statement-size <BYTES>	Abort if a single SQL statement exceeds this size (default: 256 MiB).

Performance debugging

Debug builds can print timing and throughput metrics when SKIM_DEBUG is set:

SKIM_DEBUG=1 cargo run -- dump.sql -o output.json

PowerShell:

$env:SKIM_DEBUG = "1"
cargo run -- .\dump.sql -o output.json

The debug summary is written to stderr and is compiled out for release builds.

To isolate sqlparser AST construction from row conversion and output writing, set SKIM_PROFILE_AST_ONLY. This parses matching INSERT statements into ASTs and disables output file creation:

$env:SKIM_DEBUG = "1"
$env:SKIM_PROFILE_AST_ONLY = "1"
cargo run -- --no-progress .\dump.sql -o output.json

Output formats

The format is resolved in this order:

1. --format flag (explicit override)
2. Output file extension
3. Default → JSON

Format	Flag / Extension	Notes
JSON	--format json / .json	JSON array of objects
JSONL	--format jsonl / .jsonl	One JSON object per line
CSV	--format csv / .csv	Header row + comma-separated values. NULL → empty field.
YAML	--format yaml / .yaml, .yml	Multi-document YAML, one --- document per row
TOML	--format toml / .toml	Array of tables
Parquet	--format parquet / .parquet	Requires a real file path (cannot stream to stdout)

Examples

Convert a dump to JSONL, streaming from stdin:

zcat dump.sql.gz | skim --format jsonl > rows.jsonl

Extract only the users table to CSV:

skim -t users dump.sql -o users.csv

Filter multiple tables, output to stdout as JSON:

skim -t users -t orders dump.sql

Write Parquet and hide the progress bar:

skim --no-progress dump.sql -o output.parquet

Pipe through jq:

skim dump.sql | jq '.[] | select(.active == true)'

License

MIT