Data & AI Architect · Senior Data Engineer in Kathmandu. I design data platforms, the governance around them and the AI on them, then build them: nine years in software, six on production data platforms.

• Moving analytics off the product's Postgres. Measure first, stop the damage, then scale out or migrate in stages with a way back. In production, two single-server Postgres databases, one of them a pgvector store, moved to Citus.
• Platforms chosen on evidence and run at scale. A proof of concept on criteria fixed in advance, then ingestion, lakehouse and serving that stay correct, recoverable and affordable: a 15 TB+ production footprint, 100M+ events a day, 99.9% uptime.
• Governance an auditor can check. Classification, masking, contracts and access review designed into the data model; SOC 2 controls for the platform and its infrastructure; erasure measured, not assumed.
• AI on governed data. LangGraph, Google ADK and MCP agents whose output is typed, evaluated and reviewable before it becomes state, with isolation enforced outside the model. Analytics delivery 60% faster.

Start here — two reference architectures for a scenario common across the industry, published in full and measured

• data-platform-architecture — a SaaS company leaving Postgres: the current state measured on a copy of its database at its peak rate (reports on the primary took the product's p95 from 40 to 525 ms; on the replica 3 of 35 long reports finished), options priced in its EU region, a staged target, a data model across four domains, governance with SOC 2 readiness (a 21-control matrix), migration and 18 decisions with machine-checked evidence; keeping personal data in one table cut what erasing one person rewrote from 31.8 million event rows to 60,000. A proof of concept on Databricks: the smallest warehouse ran the reports in under 3 s, a day's work took 38 s and was billed for 11 minutes awake, and Terraform owned 6 of 7 kinds of governance object. · case study
• ai-platform-architecture — the same company's usage assistant, text-to-SQL over governed data, measured on Qwen3, Llama 3.3, gpt-oss and Databricks Genie: told to stay in one customer's data, the models leaked on 23 of 120 requests; a default-deny query rewrite matched row-level security on every query they wrote and refuses 44 of 44 hand-written escapes; a semantic layer took Llama 3.3 70B from 55 to 70 of 70. · case study
• Choosing where the data lives — the decision guide behind them: RDS, Citus, TimescaleDB, ClickHouse, Trino, Iceberg and pgvector, by workload, with the measurements behind each rule.

Production use cases

Data platforms

• Rebuilding a production data platform for scale and recovery — six years on one platform: faster processing, lower cost, safer change, and recovery designed before the incident.
• An analytics lakehouse for a contact-centre platform — S3, Trino, Airflow and AWS Glue, delivered configured, with security, access control, HA and DR.

Databases

• Two Postgres servers that ran out of room, moved to Citus — an application database and a pgvector store, scaled out with the data model doing most of the work.
• Managed or self-run PostgreSQL — a capacity plan and a fleet of RDS and Aurora databases, tuned and failover-tested; the same rule decided both.

AI systems

• Turning model output into product state you can audit — a governed analytics workflow with typed state, provenance, evaluation, review and an honest fallback path.
• Short-text classification and alerting for a lakehouse team — a classifier that failed in production, rebuilt around the unit of work, on Databricks.

Open source — each with its method and measurements in the repository; case studies, grouped the same way, at basantbhattarai.com.np/work

Data platforms

• streaming-platform — Kafka, Flink, ClickHouse and Iceberg with Avro contracts and a freshness SLO; every event counted at every layer, exactly-once through broker kills, TaskManager kills and stalls. · case study
• spark-sql-kinesis — Kinesis source and sink for Spark 4.2 Structured Streaming on AWS SDK v2. Upgrades in place: the checkpoint format is byte-identical, so no drain. · case study
• dataset-registry — ownership before consumption, as a product: owners, freshness SLOs, versioned JSON Schema contracts, OpenLineage lineage, and breaking changes that wait for consumer acknowledgement. Go, PostgreSQL, Next.js. · case study
• data-stack-lab — hands-on data engineering projects, one per tool, on one generated SaaS dataset with known answers, each checked in CI. dbt: a Type 2 dimension kept all 5,048 invoices on the plan they billed, where Type 1 moved 15.1% of revenue. Airflow 3: backfilled newest-first, high-water-mark loads lost 96.56% of a month with every run green. Data quality: nine injected defects; blocking let 0 of 3,198 bad rows through, and only control totals, volume and per-platform share saw the rows that never arrived. Spark: re-running one day with the default overwrite mode left 1 of 365 daily partitions. Databricks: a truncated file corrected under the same name was skipped by COPY INTO and Auto Loader, losing 1,638 events. · case study
• lakehouse-lab — Iceberg on MinIO with Apache Polaris vending table-scoped credentials to Trino and StarRocks; dbt + Airflow medallion, compaction and rollback, every figure measured. · case study
• rag-pipeline — scrape-to-RAG lakehouse: Airflow, Spark bronze/silver/gold on Delta and MinIO, Chroma, a FastAPI answer service, OpenLineage.

AI systems

• governed-agent — an AI agent whose output becomes state only after a schema, a policy, a grounding check and, where required, a person: LangGraph + MCP, FastAPI with a PostgreSQL outbox, a Next.js review UI, measured on qwen3:8b. · case study
• evalproof — LLM evaluation as measurement: calibrate the judge against a gold sample, correct its pass rate (ACC/PPI), and gate CI only on significant regressions. Local-first with Ollama; ships as a GitHub Action. · case study · live app · pip install evalproof
• realtime-ml — real-time fraud scoring where leakage, training/serving skew and label delay are measured: Feast, LightGBM, MLflow, shadow serving, drift alerts and a paired-statistics promotion gate. · case study
• retrieval-bench — pgvector, Qdrant and OpenSearch on the same embeddings: recall–latency curves, filtered-search recall collapse and its fixes, build cost, writes under load, and hybrid vs vector nDCG on public BEIR data. · case study
• short-text-triage — label the string, not the row: multilingual embeddings, HDBSCAN and margin-banded assignment on a reproducible generated corpus; CI re-checks the figures. · case study
• prevalence-shift — why counting predicted labels flattens a trend, and how ACC, SLD and prediction-powered inference fix it; CI re-runs all 1,000 replications. · case study · live app

Databases and distributed systems

• distributed-db-lab — Postgres scale-out with Citus (migration planner, online rebalance under load with zero errors, tenant isolation), TimescaleDB at scale, and a 2×2 replicated ClickHouse cluster; every failure measured. · case study
• db-failover-lab — six database engines, eight failover scenarios on real multi-node clusters: measured HA, sharding and backup/restore for PostgreSQL 18, Redis, MongoDB, ScyllaDB, SolrCloud and Galera. · case study
• postgres-ha-lab — Patroni + etcd on amd64 Rocky 8 under QEMU and pgBackRest over mutual TLS: failover, replica rebuild and point-in-time restore actually performed. · case study

Delivery and infrastructure

• platform-deployments — GitOps for governed-agent on a three-node Kubernetes cluster: Argo CD, an Argo Rollouts canary that aborted a bad release on its own in 6 of 6 runs, CloudNativePG, Kyverno and OpenTelemetry, plus Terraform for EKS. · case study

Upstream

• awslabs/spark-sql-kinesis-connector#99 — an opt-in option to stop reading a shard once it is caught up, requested in the project's issue #11 (in review).
• citusdata/citus#7333 — a minimal reproduction on Citus 14.2 of node removal leaving cleanup records behind, with the code path; an earlier report of mine, #4776, led to unnamed-constraint support on distributed tables.

Recent writing — measured, with the method stated; all 50 pieces, grouped by subject

• RDS, Citus, TimescaleDB, ClickHouse, Trino, Iceberg or pgvector: choosing where the data lives
• Erasing one person rewrote 31.8 million rows. With their e-mail in one table, 60,000.
• The day's work took 38 seconds. The warehouse was billed for 11 minutes.
• We deleted the user from the Delta table. Time travel still returned all 100 rows.
• Told to stay in one customer's data, the models leaked 23 times in 120
• For text-to-SQL, the definitions mattered more than the model
• The read replica was meant to take the reports. 3 of 35 finished.

Stack

Python · SQL · Go · TypeScript · Scala · Spark · Flink · Kafka · Kinesis · Airflow · dbt · Databricks · Unity Catalog · DuckDB · Iceberg · Trino · ClickHouse · PostgreSQL · Citus · pgvector · Kubernetes · Argo CD · Terraform · OpenTofu · AWS · LangGraph · MCP · MLflow · Feast · Next.js

basantbhattarai.com.np · Résumé · Work with me · LinkedIn · contact@basantbhattarai.com.np