| Engine / Repository | Pull Request | Technical Scope | Status |
|---|---|---|---|
| NVIDIA/cudf-spark | #16863 | Fix buffer and resource leaks on multithreaded file reader cancellation [re... | Open (In Review) · 2026-11-19 |
| NVIDIA/cudf-spark | #16881 | Fix multithreaded read when maxNumFilesParallel is below task file count [r... | Open (In Review) · 2026-11-19 |
| databricks/dbt-databricks | #1767 | feat: supplement delta.dataSkippingStatsColumns for snapshot tables (#1481) | Open (In Review) · 2026-11-16 |
| ibis-project/ibis | #12214 | fix(postgres): read query schema without temporary view DDL for read-only s... | Open (In Review) · 2026-11-16 |
| apache/iceberg-python | #4265 | Fix: assert ref snapshot id requirements during snapshot expiration (#4200) | Open (In Review) · 2026-11-12 |
| delta-io/delta-rs | #5028 | feat(python): support tuple filters in write, update, and delete predicates... | Open (In Review) · 2026-11-08 |
| databricks/dbt-databricks | #1766 | feat: allow passing query tags via environment variables (#1454) | Open (In Review) · 2026-11-06 |
| ibis-project/ibis | #12211 | fix(expr): type decimal mean as float64 and decimal product with combined s... | Open (In Review) · 2026-11-03 |
|
Medallion Lakehouse on Databricks featuring Auto Loader, Delta CDF, SCD1/SCD2, Great Expectations quarantine gates, offline SQLGlot AST eval (~85ms), and AutoHeal-DE partition recovery.
|
In-memory vectorized columnar query execution engine exploring contiguous
|
Open-source static AST optimizer, diagnostic engine, and built-in MCP server (
|
flowchart LR
Ingest[Auto Loader / Kafka] --> Compute[Spark / RAPIDS GPU]
Compute --> Gates{Quality & AST Gates}
Gates -->|Valid| Storage[(Delta / Iceberg · Unity Catalog)]
Gates -->|Quarantine| AutoHeal[AutoHeal-DE Partition Merge]
Storage --> Serve[Databricks Genie / MCP Server]
- Legacy Data Modernization (
2025-03) — Cloud-native lake, Glue ETL, and Redshift analytics - Productionizing an ML-Driven Allocation Engine (
2025-02) — End-to-end ML allocation — eligibility, classification, and production serving on AWS - Automated Web Intelligence Pipeline (
2025-01) — Crawlers, extraction, deduplication, and DS model integration on AWS - From OLAP to Workload-Specific Architecture (
2024-12) — Production migration — Redshift cost signal, engine benchmarks, workload separation

