Skip to content
codecapoPublic

About

Open-source data platform in Rust: medallion pipelines on Apache Iceberg, DataFusion and Arrow

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Infostash

CI CodeQL codecov OpenSSF Scorecard License

An open-source data platform written in Rust. Load data, clean it, and turn it into business-ready tables with plain SQL, then ask questions of it in plain English. It runs on AWS, GCP, Azure or your own servers, stores everything in open formats in your own storage, and shuts compute off when nobody is using it.

Status: pre-alpha. We are building the first working slice. Nothing here is ready for production. Star or watch the repo to follow along, and see ROADMAP.md for what comes next.

What it does

  • Medallion pipelines: data moves through three layers. Raw is the data as it arrived, Clean is typed, deduplicated and checked, and Ready holds the tables reports use. Each layer is a folder of SQL files.
  • Incremental by default: each run processes only the rows that changed.
  • Bad rows don't break runs: rows that fail checks go to a quarantine table.
  • Open tables: everything is stored as Apache Iceberg on object storage, so Spark, Trino, Snowflake and Databricks can read the same tables.
  • Scales to zero: compute runs only while there is work to do.
  • Agents: a chat agent answers questions from defined business metrics, and a builder agent drafts pipelines on a branch for a person to approve.
  • Open connectors: write a source in YAML, Rust or any language via Wasm.

Infostash architecture

How it fits together: docs/architecture.md.

Quick start (planned for the first release)

infoctl init my-project
cd my-project
infoctl run
infoctl query "select * from gold.daily_revenue"

Build from source

You need rustup. The exact Rust version is pinned in rust-toolchain.toml, and rustup installs it automatically.

git clone https://github.com/codecapo/infostash
cd infostash
cargo build
cargo test

Get involved

License

Apache License 2.0.

About

Open-source data platform in Rust: medallion pipelines on Apache Iceberg, DataFusion and Arrow

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages