Bill Split Quest

Gamified bill splitting app with a classical computer vision + OCR pipeline for real-world receipts.

• Developed a full-stack bill-splitting application that automatically extracts bill details from receipt images.
• Built a computer vision pipeline using OpenCV for receipt detection, perspective correction, image enhancement, and deskewing.
• Integrated Tesseract OCR and Gemini AI to extract and structure receipt information such as items, subtotal, tax, and total.
• Implemented intelligent orientation detection and fallback mechanisms for robust real-world receipt processing.
• Enabled automated expense sharing by assigning items to users and calculating individual payment amounts.

Tech Stack: Python, FastAPI, React, OpenCV, Tesseract OCR, Gemini AI.

Quick Start

cd /home/pritha/dip_proj

# Backend
python -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
sudo apt-get update && sudo apt-get install -y tesseract-ocr
./.venv/bin/python -m uvicorn backend.app:app --host 127.0.0.1 --port 8000

Open a second terminal:

cd /home/pritha/dip_proj/frontend
npm install
npm run dev -- --host 127.0.0.1 --port 5173

Then open http://127.0.0.1:5173.

This project currently includes:

• Edge-first receipt preprocessing pipeline with stage-by-stage debug outputs
• OCR + table-aware bill parsing (items, subtotal, tax, grand total)
• Interactive frontend to review extraction, assign items to players, and compute split totals

Tech Stack

• Frontend: React + Vite
• Backend: FastAPI + OpenCV + pytesseract
• OCR Engine: Tesseract (tesseract-ocr system package)

Repository Layout

• backend/app.py: FastAPI app and API endpoints
• backend/dip/receipt_ocr_pipeline.py: standalone modular OCR preprocessing pipeline
• backend/dip/: legacy DIP helpers used by API overlays/segmentation
• backend/ocr/extractor.py: OCR helper functions (full-line and line-by-line)
• backend/utils/parser.py: table-aware parsing and debug decisions
• backend/utils/tax.py: equal/proportional/ignore tax split logic
• frontend/src/: multi-screen gamified UI
• BillSplitQuest_PRD.md: product requirements document

Basic Requirements

• Linux/macOS/Windows
• Python 3.10+
• Node.js 18+
• npm 9+
• Tesseract OCR installed and available in PATH

Install Tesseract (Ubuntu)

sudo apt-get update
sudo apt-get install -y tesseract-ocr

Setup

1. Backend

From project root (/home/pritha/dip_proj):

python -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
./.venv/bin/python -m uvicorn backend.app:app --host 127.0.0.1 --port 8000

Backend API: http://127.0.0.1:8000

Health check:

curl http://127.0.0.1:8000/health

2. Frontend

cd frontend
npm install
npm run dev -- --host 127.0.0.1 --port 5173

Frontend URL: http://127.0.0.1:5173

Optional backend override:

VITE_API_URL=http://127.0.0.1:8000 npm run dev

Current Processing Pipeline (API)

POST /process-bill performs:

1. Receipt preprocessing via receipt_ocr_pipeline.py
2. Final OCR-ready image generation
3. Horizontal line segmentation
4. Line-by-line OCR
5. Table-aware parsing into:
   • items
   • subtotal (before tax)
   • tax
   • total (grand total)

The API also returns debugging payloads:

• stages: base64 images for intermediate pipeline stages
• ocr: candidate/confidence/raw lines/line-by-line output
• parse_debug: accepted and rejected parse decisions
• pipeline_meta: pipeline metadata (document detection/skew/line count)

API Endpoints

POST /process-bill

Input: multipart/form-data with image

Output (high-level):

{
   "items": [{"name": "...", "price": 0}],
   "subtotal": 0,
   "tax": 0,
   "total": 0,
   "ocr": {...},
   "parse_debug": {...},
   "pipeline_meta": {...},
   "stages": {...}
}

POST /calculate-split

Input:

{
   "items": [{"name": "...", "price": 0}],
   "assignments": {"Item A": ["Alice", "Bob"]},
   "tax_mode": "equal|proportional|ignore",
   "tax": 0
}

Output:

{
   "Alice": 120.5,
   "Bob": 98.3
}

Standalone Pipeline Debug Runner

You can run preprocessing without the web app:

cd /home/pritha/dip_proj
./.venv/bin/python backend/dip/receipt_ocr_pipeline.py backend/dip/img1.jpeg \
   --output-dir pipeline_debug \
   --block-size 15 \
   --c 3 \
   --skew \
   --lines

Optional tuning flags:

• --canny-low
• --canny-high
• --min-receipt-area-ratio

Notes and Known Behavior

• Receipt extraction is robust but may still need threshold tuning for low-contrast or cluttered backgrounds.
• Parser is table-aware, but OCR quality still strongly affects final item accuracy.
• Grand total is treated as tax-inclusive; subtotal is pre-tax.
• Assignment step enforces at least one player per item before split calculation.

Troubleshooting

• Could not read image: check image path.
• TesseractNotFoundError: install tesseract-ocr and ensure it is in PATH.
• Blank frontend: ensure backend/frontend are both running and check browser console.
• Unexpected totals: inspect ocr.lines and parse_debug in Item Reveal to see exactly what was accepted/rejected.