data-skew

Databricks PySpark Certification Prep Lab: Build an e-commerce analytics pipeline covering Spark DataFrame API, Structured Streaming, data skew handling with salting, broadcast joins, and Pandas UDFs. Designed for the Databricks Certified Associate Developer for Apache Spark exam.

Agent Skill for optimizing Apache Hive on MapReduce — 26 actionable best-practice rules covering storage, query, JOIN, data skew, and MR parameter tuning. Works with Cursor & Claude Code.

Didactic simulations to master Apache Spark. Learn Query Plans, DAGs, Data Skew resolution, and performance tuning with hands-on exercises and real financial data.

Finds the skew that makes your Spark jobs list. Reads the event logs Spark already writes: stragglers with their cause (hot key vs slow node), spill, GC pressure, priced in drag seconds. No Spark install needed; 600MB logs parsed in 4s.

Agent Skill for optimizing Apache Hive on MapReduce — 26 actionable best-practice rules covering storage, query, JOIN, data skew, and MR parameter tuning. Works with Cursor & Claude Code.

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning

Instant diagnostics for Pandas and PySpark DataFrames: detect schema drift, null spikes, duplicates, skew, risky joins, and pipeline health issues.

Diagnoses Spark data skew from the job's own event log - no cluster, no JVM - and tells key skew apart from a straggler by comparing task time against bytes read. Then measures the fixes against each other on real Spark 4: across 16 cells, salting won 0. Broadcast 6.5x, AQE 1.5x, salting 0.81x.