🛡️ lake-guard

Modern Schema Drift & Data Contract CI/CD Guard for Lakehouses and Data Pipelines

Catch breaking schema changes, invalid data types, and contract violations in your CI/CD pipeline before they reach production.

💡 Why lake-guard?

In modern data stacks, upstream services and ETL pipelines constantly evolve. Without automated contract enforcement:

Backend teams drop or rename columns, crashing downstream Spark/dbt pipelines.
Data types change silently (e.g. BIGINT to INT, or numeric strings to numbers), causing truncation or overflow.
Non-nullable columns suddenly receive NULL values, breaking BI dashboards.
Pull Requests lack visibility into data distribution shifts.

lake-guard is a lightweight, zero-cloud developer tool and CI/CD bot powered by DuckDB and PyArrow. It compares Lakehouse snapshots (Parquet, Delta Lake, CSV) against declared contracts or base branches, generating instant PR reports and failing CI builds when breaking changes are detected.

⚡ Architecture

flowchart TD
    subgraph Input ["Data Sources"]
        A["Base / Prod Dataset (Parquet / Delta / CSV)"]
        B["Target / PR Dataset (Parquet / Delta / CSV)"]
        C["Data Contract (YAML)"]
    end

    subgraph Core ["lake-guard Engine"]
        D["DuckDB Profiler & PyArrow Scanner"]
        E["Type Compatibility & Widening Rules"]
        F["Contract & Constraint Validator"]
    end

    subgraph Outputs ["CI / CD Outputs"]
        G["Rich Terminal Output"]
        H["GitHub PR Sticky Comment (Markdown)"]
        I["JSON Machine-Readable Report"]
        J["Exit Code 0 (Pass) / Exit Code 1 (Block CI)"]
    end

    A --> D
    B --> D
    C --> F
    D --> E
    E --> F
    F --> G
    F --> H
    F --> I
    F --> J

✨ Features

⚡ Blazing Fast & Zero-Cloud: Runs locally or inside GitHub Actions without needing Snowflake, Databricks, or external infrastructure. Powered by in-memory DuckDB.
📜 Declarative Data Contracts: Define column types, nullability, uniqueness, value ranges, allowed enums, regex patterns, and row count bounds in clean YAML.
🔍 Smart Schema Drift Detection: Distinguishes between:
- 🔴 BREAKING: Dropped columns, integer narrowing (INT8 -> INT4), non-null violations, enum violations.
- 🟡 WARNING: Significant row count drops, null-rate spikes (>10%).
- 🟢 SAFE: Backwards-compatible alterations, safe type widening (INT4 -> INT8, DATE -> TIMESTAMP), new nullable columns.
🤖 Native GitHub PR Integration: Generates formatted GitHub Markdown tables and status badges ready for PR comment bots.
🪄 Auto-Contract Generation: Bootstrap a contract from any existing dataset in 1 second with lakeguard generate.

📦 Installation

Using `uv` (Recommended)

uv pip install lake-guard

Using `pip`

pip install lake-guard

From source

git clone https://github.com/src-id/lake-guard.git
cd lake-guard
uv sync

🚀 Quickstart Guide

1. Initialize lake-guard in your repository

lakeguard init

This generates a starter lakeguard_contract.yaml and .github/workflows/lakeguard_pr.yml.

2. Auto-generate a contract from an existing table

lakeguard generate ./data/orders.parquet --output ./contracts/orders_contract.yaml

3. Validate a dataset against your contract

lakeguard check ./data/staging_orders.parquet --contract ./contracts/orders_contract.yaml

Output:

╭────────── ✔ Data Contract Passed: orders ───────────╮
│ Status:  PASSED                                     │
│ Target: ./data/staging_orders.parquet               │
│ Total Rows: 1,000                                   │
│ Breaking Issues: 0  |  Warnings: 0  |  Safe/Info: 2 │
╰─────────────────────────────────────────────────────╯
                          Contract Evaluation Details
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Column        ┃ Rule / Change Type ┃ Details                      ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 🟢 SAFE  │ notes         │ column_added       │ Additional column 'notes'    │
│          │               │                    │ detected (allowed by contract│
│ 🟢 SAFE  │ discount_rate │ column_added       │ Additional column 'discount' │
└──────────┴───────────────┴────────────────────┴──────────────────────────────┘

4. Diff two dataset snapshots directly (Base vs Target)

lakeguard diff ./data/production.parquet ./data/staging.parquet

If breaking changes are detected:

╭─ ✖ Schema Diff Violated: Breaking Changes Detected ─╮
│ Status:  FAILED                                     │
│ Base:   ./data/production.parquet                   │
│ Target: ./data/staging.parquet                      │
│ Total Rows: 1,000 ➔ 1,000 (+0.0%)                   │
│ Breaking Issues: 2  |  Warnings: 0  |  Safe/Info: 0 │
╰─────────────────────────────────────────────────────╯
                         Schema Diff Evaluation Details                         
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃  Severity   ┃ Column       ┃ Rule / Change Type ┃ Details                    ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 🔴 BREAKING │ customer_id  │ column_removed     │ Column 'customer_id' was   │
│             │              │                    │ removed from target dataset│
│ 🔴 BREAKING │ order_amount │ type_mismatch      │ Incompatible type          │
│             │              │                    │ alteration: DECIMAL(22,1)  │
│             │              │                    │ -> DECIMAL(23,2)           │
└─────────────┴──────────────┴────────────────────┴────────────────────────────┘

(Exit code is 1, automatically blocking the CI pipeline).

📝 Contract Specification Example

name: orders_lakehouse
description: Production orders dataset contract
version: 1.0.0
min_rows: 100
max_rows: 1000000
allow_extra_columns: true

columns:
  - name: order_id
    data_type: INT8
    nullable: false
    unique: true
    description: Primary order identifier

  - name: customer_id
    data_type: VARCHAR
    nullable: false
    pattern: "^CUST-[0-9]{5}$"

  - name: order_amount
    data_type: FLOAT8
    nullable: false
    min_value: 0.01
    max_value: 100000.00

  - name: status
    data_type: VARCHAR
    nullable: false
    allowed_values:
      - PENDING
      - SHIPPED
      - DELIVERED
      - CANCELLED

  - name: created_at
    data_type: TIMESTAMP
    nullable: false

🤖 GitHub Actions CI Integration

Automatically validate data changes in every Pull Request and post summary comments:

name: Data Contract CI Guard

on:
  pull_request:
    paths:
      - 'data/**'
      - 'contracts/**'

jobs:
  contract-check:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v3

      - name: Install lake-guard
        run: uv pip install lake-guard

      - name: Audit Data Contract
        id: audit
        run: |
          uv run lakeguard check ./data/staging.parquet \
            --contract ./contracts/orders_contract.yaml \
            --format markdown \
            --output pr_comment.md

      - name: Post Comment to PR
        uses: marocchino/sticky-pull-request-comment@v2
        if: always()
        with:
          header: lake-guard-report
          path: pr_comment.md

Example PR Comment:

Data Contract Verification

Contract: orders_lakehouse
Target: staging.parquet
Total Rows: 1,000

Severity Count

🔴 Breaking Issues 0

🟡 Warnings 0

🟢 Safe / Info 2

Generated by lake-guard

🛠️ CLI Command Reference

Command	Arguments / Flags	Description
`lakeguard check`	`<target> -c <contract.yaml> [--fail-on breaking\|warning] [-f console\|markdown\|json] [-o out.md]`	Validate dataset against a contract specification
`lakeguard diff`	`<base> <target> [--null-threshold 10.0] [--row-threshold 20.0] [-f console\|markdown\|json] [-o out.md]`	Diff two datasets directly for schema & distribution shifts
`lakeguard generate`	`<dataset> [-o contract.yaml] [-t table_name]`	Auto-generate contract YAML from an existing dataset
`lakeguard init`	—	Scaffold starter contract and GitHub Actions workflow
`lakeguard version`	—	Display CLI version and DuckDB/PyArrow engine info

🧪 Development & Testing

# Clone the repository
git clone https://github.com/src-id/lake-guard.git
cd lake-guard

# Install dependencies
make install

# Generate demo datasets
make demo

# Run the test suite
make test

# Run interactive demo checks
make check-demo
make diff-demo

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'feat: add support for Iceberg REST catalog')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

Distributed under the MIT License. See LICENSE for more information.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
examples		examples
src/lake_guard		src/lake_guard
tests		tests
.gitignore		.gitignore
.python-version		.python-version
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Navigation Menu

Uh oh!

Repository files navigation

🛡️ lake-guard

💡 Why lake-guard?

⚡ Architecture

✨ Features

📦 Installation

Using `uv` (Recommended)

Using `pip`

From source

🚀 Quickstart Guide

1. Initialize lake-guard in your repository

2. Auto-generate a contract from an existing table

3. Validate a dataset against your contract

4. Diff two dataset snapshots directly (Base vs Target)

📝 Contract Specification Example

🤖 GitHub Actions CI Integration

Example PR Comment:

Data Contract Verification

🛠️ CLI Command Reference

🧪 Development & Testing

🤝 Contributing

📄 License

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

🛡️ lake-guard

💡 Why lake-guard?

⚡ Architecture

✨ Features

📦 Installation

Using uv (Recommended)

Using pip

From source

🚀 Quickstart Guide

1. Initialize lake-guard in your repository

2. Auto-generate a contract from an existing table

3. Validate a dataset against your contract

4. Diff two dataset snapshots directly (Base vs Target)

📝 Contract Specification Example

🤖 GitHub Actions CI Integration

Example PR Comment:

Data Contract Verification

🛠️ CLI Command Reference

🧪 Development & Testing

🤝 Contributing

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Using `uv` (Recommended)

Using `pip`