Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

14 Commits

Folders and files

Repository files navigation

Sahaayika (सहायिका) 🌸

Multimodal AI Companion & Visual Grounding Guide for Digital Public Infrastructure

Gemini Android FastAPI Render Docker License: MIT

"दीदी, चिंता मत करो, मैं हर कदम पर आपके साथ हूँ।"
(Sister, do not worry, I am with you at every step.)


📖 Overview

Sahaayika (सहायिका) is a multimodal, vernacular AI assistant designed to bridge India's digital divide. Millions of rural citizens—especially women and first-time smartphone users—are structurally excluded from life-changing welfare schemes (such as Ladli Behna Yojana, PM-Kisan, Ration Card, and Janani Suraksha) because government portals are dense, unlabelled, and filled with bureaucratic jargon.

Traditional text-prompting chatbots fail these users. Sahaayika flips the paradigm to Proactive Visual & Audio Guidance:

  1. Speaks First in Vernacular: Immediately greets and guides in native regional languages (Hindi, Tamil, Bengali, Telugu, Marathi, English).
  2. Visual Spotlight Aperture: Ingests browser screens and uses Google Gemini 3.8 Flash to physically highlight exact input fields with a glowing gold spotlight on top of the live web browser.
  3. Empathetic Companion Persona ("Didi"): Answers citizen doubts in simple, encouraging terms without confusing technical jargon.

🚀 Live Cloud Deployment

The Sahaayika Brain backend is deployed live on Render:


📱 Mobile App (Android Client) - v1.1.0 (Optimized ~3.09 MB)

The APK size has been reduced by 81% (from 16.3 MB down to ~3.09 MB) using full R8 dead-code stripping, resource shrinking, and dual CPU architecture specialization:

📦 View GitHub Release v1.1

Required Android Permissions:

  1. Accessibility Service (SahaayikaAccessibilityService): Enables passive screenshot capture when navigating scheme portals to compute grounding coordinates without breaking user flow.
  2. Display Over Other Apps (TYPE_APPLICATION_OVERLAY): Renders the floating "Didi" avatar and the dynamic visual spotlight cutout over web browsers.
  3. Microphone (RECORD_AUDIO): Enables vernacular voice search and intent resolution.

🏛️ System Architecture

┌─────────────────────────────────────────────────────────────────┐
│               Android Thin Client (Edge / Kotlin)               │
│                                                                 │
│  [LanguageActivity] ──> [GuidedHubActivity] ──> [Scheme Portal] │
│          │                       │                     │        │
│    (Vernacular TTS)       (Voice Intent Mic)   (Chrome / Web)   │
│                                                        │        │
│  ┌───────────────────────┐             ┌───────────────▼──────┐ │
│  │ FloatingDidiAvatar    │             │ SahaayikaA11yService │ │
│  │ (Draggable & Speech)  │             │ (Silent Snapshot)    │ │
│  └───────────▲───────────┘             └───────────────┬──────┘ │
│              │                                         │        │
│  ┌───────────┴───────────┐                             │        │
│  │ SpotlightCanvasView   │◄────────[box_2d & TTS]──────┤        │
│  │ (Clear Aperture + Ring)│                             │        │
│  └───────────────────────┘                             │        │
└────────────────────────────────────────────────────────┼────────┘
                                                         │
                                        HTTPS / REST API │ (Multipart / JSON)
                                                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                 FastAPI Backend (Cloud / Render)                │
│                                                                 │
│  • POST /api/v1/intent         ──> Scheme matching engine       │
│  • POST /api/v1/analyze-screen ──> Gemini 3.8 Flash Vision      │
│  • POST /api/v1/didi/ask       ──> Empathetic Conversational QA │
│  • GET  /api/v1/schemes        ──> Multilingual catalog         │
│                                                                 │
│                     ┌────────────────────┐                      │
│                     │  Google Gemini API │                      │
│                     │ (gemini-3.8-flash) │                      │
│                     └────────────────────┘                      │
└─────────────────────────────────────────────────────────────────┘

✨ Key Features & Technical Highlights

1. Multimodal Visual Grounding via Gemini 3.8 Flash

  • Uses high-speed screen understanding with normalized box_2d coordinates: [ymin, xmin, ymax, xmax].
  • The Android client translates normalized coordinates into device physical pixels via SpotlightCanvasView.
  • Clears a transparent portal aperture using PorterDuff.Mode.CLEAR and animates a pulsing 24K gold border around the active field.

2. Zero-Prompting Voice Intent Hub

  • Citizens simply tap the glowing microphone button and speak freely in their native dialect (e.g., "मेरी लाडली बहना के पैसे देखने हैं").
  • The intent pipeline resolves the goal, suggests matching official schemes, and automatically launches the official government portal.

3. Native Indic Audio Pipeline

  • Integrates IndicTextToSpeech with regional locales (hi-IN, mr-IN, ta-IN, te-IN, bn-IN, en-IN).
  • Automatically narrates every step and guides first-time users before they even need to ask.

4. Privacy-by-Design Local Scrim

  • Passively monitors UI element types to mask sensitive password or financial inputs before screenshot dispatch.
  • Debounced accessibility event queue (1500ms) with concurrency mutex prevents duplicate network calls.

🛠️ Local Development & Running

Prerequisites

  • Python 3.11+
  • Android Studio / Android SDK 34 (Java 17+)
  • Docker & Docker Compose (Optional)
  • Google Gemini API Key (Get one here)

Backend Setup (FastAPI)

  1. Navigate to the backend directory:

    cd backend
  2. Create and activate a virtual environment:

    python -m venv .venv
    # Windows:
    .venv\Scripts\activate
    # macOS/Linux:
    source .venv/bin/activate
  3. Install dependencies:

    pip install -r requirements.txt
  4. Configure environment variables: Create a .env file in backend/:

    GEMINI_API_KEY=your_gemini_api_key_here
    GEMINI_MODEL=gemini-3.8-flash
    DEFAULT_LANGUAGE=hi-IN
    PORT=8000
  5. Start the server:

    uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
  6. Run Backend Test Suite:

    pytest -v
    # 71 passed in ~5.5s

Docker Deployment

To run the containerized backend locally:

docker compose up -d --build

Health check verification:

curl http://localhost:8000/health
# {"status":"ok","service":"sahaayika-brain","version":"1.0.0"}

Android Client Setup

  1. Open the android folder in Android Studio.
  2. Sync Gradle dependencies.
  3. Verify or update backend URL in SahaayikaApiService.kt:
    // For live cloud:
    const val DEFAULT_BASE_URL = "https://sahaayika-brain.onrender.com/api/v1"
    // For local emulator:
    // const val DEFAULT_BASE_URL = "http://10.0.2.2:8000/api/v1"
  4. Build and run unit tests:
    cd android
    ./gradlew testDebugUnitTest
    # 27 tests passed
  5. Build the debug APK:
    ./gradlew assembleDebug
    # Output: android/app/build/outputs/apk/debug/app-debug.apk

🧪 Verification & Test Coverage

Component Test Suite Tests Result
Backend API pytest tests/ 71 Tests ✅ Passing
Android Client testDebugUnitTest 27 Tests ✅ Passing
E2E Visual Grounding Live Android 14 Emulator Tested ✅ Aperture Drawn
Audio Vocalization IndicTextToSpeech Tested ✅ Vernacular Audio

📜 License

This project is licensed under the MIT License.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages