⚡ Ten self-contained, 10-minute tutorials that walk you from zero to a modern data stack: Postgres → dbt → Great Expectations → MLflow → Kafka/Spark → Prefect → Grafana. Clone any folder, run docker compose up, and you’re exploring in minutes.
A collection of 10-minute demos showcasing modern data engineering tools and practices.
| No. | Topic | Status | Description |
|---|---|---|---|
| 01 | Postgres + dbt quick-start | ✅ Complete | Data transformation with dbt |
| 02 | Great Expectations with S3 | ✅ Complete | Data quality validation |
| 03 | FastAPI + MLflow tracking | ✅ Complete | ML model serving & tracking |
| 04 | Airbyte to DuckDB | ✅ Complete | Data integration & warehousing |
| 05 | Streamlit → Grafana | ✅ Complete | Real-time metrics & visualization |
| 06 | Kafka + Spark Structured Streaming | ✅ Complete | Real-time streaming |
| 07 | Superset + SQLite | ✅ Complete | Business intelligence |
| 08 | Prefect 3 local deployment | ✅ Complete | Workflow orchestration |
| 09 | Dagster data assets | ✅ Complete | Asset-based orchestration & lineage |
| 10 | Snowflake + dbt Cloud | ✅ Complete | Cloud data warehouse & transformation |
Each demo is self-contained and can be run independently:
# Navigate to any demo
cd 01-postgres-dbt-quickstart
# Follow the README instructions| Category | Tools |
|---|---|
| Databases | PostgreSQL, DuckDB, SQLite, Snowflake |
| ETL/ELT | dbt, Airbyte, Prefect, Dagster |
| Data Quality | Great Expectations |
| ML/AI | MLflow, BentoML, spaCy |
| Streaming | Kafka, Spark |
| Visualization | Grafana, Superset |
| Infrastructure | Docker, Kubernetes |
These demos follow a logical progression:
- Data Storage (Postgres, DuckDB, Snowflake)
- Data Transformation (dbt, Airbyte)
- Data Quality (Great Expectations)
- ML Pipeline (MLflow, BentoML)
- Streaming (Kafka, Spark)
- Orchestration (Prefect, Dagster)
- Visualization (Grafana, Superset)
Feel free to contribute additional demos or improvements!
- GitHub: @Kalyan1210
- Email: yalla.saikalyan@gmail.com
- LinkedIn: Sai Kalyan