🤖 Autolysis — Automated Data Analysis & AI Narrative Engine

TDS Project 2 · Drop in any CSV. Get statistics, visualizations, and an AI-generated narrative — fully automated.

---

📌 Table of Contents

• Overview
• How It Works
• Repository Structure
• Datasets Analyzed
• Pipeline Breakdown
• Getting Started
• Output Files
• Technical Architecture
• Sample Results

---

🧭 Overview

Autolysis is a zero-configuration data analysis pipeline. Pass it any CSV file and it will:

1. Load & clean the data (handles encoding quirks, missing values, type inference)
2. Analyze it — summary statistics, correlations, and IQR-based outlier detection
3. Visualize — correlation heatmaps, outlier bar charts, and distribution plots saved as PNGs
4. Narrate — calls an LLM (GPT-4o-mini via AI Proxy) to generate a written story from the findings
5. Report — outputs a fully-formed README.md with embedded charts and narrative

No configuration files. No manual steps. One command.

python autolysis.py dataset.csv

---

⚙️ How It Works

CSV Input
    │
    ▼
┌─────────────────────┐
│  Data Loading        │  pandas · ISO-8859-1 encoding support
└──────────┬──────────┘
           │
    ▼
┌─────────────────────┐
│  Statistical         │  describe() · isnull() · .corr()
│  Analysis            │
└──────────┬──────────┘
           │
    ▼
┌─────────────────────┐
│  Outlier Detection   │  IQR method (Q1 - 1.5×IQR / Q3 + 1.5×IQR)
└──────────┬──────────┘
           │
    ▼
┌─────────────────────┐
│  Visualization       │  seaborn · matplotlib → PNG files
└──────────┬──────────┘
           │
    ▼
┌─────────────────────┐
│  LLM Narrative       │  GPT-4o-mini via AIPROXY_TOKEN
└──────────┬──────────┘
           │
    ▼
┌─────────────────────┐
│  README Generation   │  Markdown report with embedded visuals
└─────────────────────┘

---

📁 Repository Structure

Tools-In-Data-Science-Project-2/
│
├── autolysis.py              # Core analysis engine (single-file, self-contained)
│
├── goodreads/                # Analysis output — Goodreads dataset
│   ├── README.md
│   ├── correlation_matrix.png
│   ├── outliers.png
│   └── distribution_*.png
│
├── happiness/                # Analysis output — World Happiness dataset
│   ├── README.md
│   ├── correlation_matrix.png
│   ├── outliers.png
│   └── distribution_*.png
│
├── media/                    # Analysis output — Media consumption dataset
│   ├── README.md
│   ├── correlation_matrix.png
│   ├── outliers.png
│   └── distribution_*.png
│
├── LICENSE                   # MIT
└── README.md                 # You are here

---

📊 Datasets Analyzed

📚 Goodreads

Book ratings, author profiles, genre distribution, and publication trends. Key questions: What makes a book highly rated? Which authors dominate by volume vs. quality?

🌍 Happiness (World Happiness Report)

Country-level happiness scores correlated against GDP, social support, health, freedom, and corruption. Key questions: What factors most strongly predict national happiness?

🎬 Media

Media consumption data — views, ratings, and genre breakdown. Key questions: Are certain genres consistently better rated? What does engagement look like across media types?

---

🔬 Pipeline Breakdown

1. analyze_data(df)

Runs df.describe() for summary stats, counts nulls per column, and computes a full Pearson correlation matrix on all numeric columns.

2. detect_outliers(df)

Uses the IQR method — values beyond Q1 − 1.5×IQR or Q3 + 1.5×IQR are flagged as outliers, counted per column.

3. visualize_data(...)

Generates three charts per dataset:

• Correlation Heatmap — Blues palette, annotated cells
• Outlier Bar Chart — only rendered when outliers are detected
• Distribution Plot — KDE-smoothed histogram of the first numeric column

4. question_llm(prompt, context)

Posts to the AI Proxy endpoint (aiproxy.sanand.workers.dev) with GPT-4o-mini. Passes summary stats and missing value counts as context. Returns a narrative string appended to the report.

headers = {"Authorization": f"Bearer {token}"}
data = {"model": "gpt-4o-mini", "messages": [...], "max_tokens": 1000}
response = requests.post(api_url, headers=headers, data=json.dumps(data))

5. create_readme(...)

Assembles a Markdown report with embedded PNG references, summary stat tables (via .to_markdown()), and the LLM narrative.

---

🚀 Getting Started

Prerequisites

Dependencies are declared inline via PEP 723 script metadata and can be installed with:

pip install pandas seaborn matplotlib numpy scipy openai scikit-learn requests

Environment Variable

The LLM narrative step requires an AI Proxy token:

export AIPROXY_TOKEN=your_token_here

The tool uses https://aiproxy.sanand.workers.dev — a cost-efficient proxy to OpenAI's API. Without a token, the analysis still runs; only the narrative generation step fails gracefully.

Run

python autolysis.py path/to/your/dataset.csv

Output is written to an output/ directory created automatically in the current working directory.

---

📤 Output Files

File	Description
README.md	Full analysis report with embedded charts and LLM narrative
correlation_matrix.png	Heatmap of Pearson correlations between numeric columns
outliers.png	Bar chart of outlier counts per column (if any found)
distribution_<col>.png	KDE histogram of the first numeric column

---

🏗 Technical Architecture

Component	Technology
Data loading	pandas (ISO-8859-1 encoding)
Statistics	pandas, numpy
Outlier detection	IQR method via numpy
Visualizations	seaborn, matplotlib
LLM integration	OpenAI GPT-4o-mini via AI Proxy
Report generation	Python file I/O + Markdown
CLI	argparse / sys.argv
Dependency spec	PEP 723 inline script metadata

---

📋 Sample Results

Happiness Dataset — Notable Correlation: GDP per capita and Life Ladder (happiness score) show strong positive correlation, confirming economic prosperity as a primary driver of reported well-being across nations.

Goodreads Dataset — Notable Finding: Books with higher ratings tend to have more text reviews than star-only ratings, suggesting engaged readers leave qualitative feedback more than disengaged ones.

Media Dataset — Notable Finding: Overall quality ratings cluster tightly regardless of language, suggesting audience satisfaction is driven more by content quality than production origin.

---

👤 Author

Aman Mani Tiwari github.com/AmanManiTiwari

---

📄 License

MIT — see LICENSE for details.

---

Built as part of the Tools in Data Science course. Analysis powered by the GitHub API, OpenAI GPT-4o-mini, and standard Python data science libraries.