"दीदी, चिंता मत करो, मैं हर कदम पर आपके साथ हूँ।"
(Sister, do not worry, I am with you at every step.)
Sahaayika (सहायिका) is a multimodal, vernacular AI assistant designed to bridge India's digital divide. Millions of rural citizens—especially women and first-time smartphone users—are structurally excluded from life-changing welfare schemes (such as Ladli Behna Yojana, PM-Kisan, Ration Card, and Janani Suraksha) because government portals are dense, unlabelled, and filled with bureaucratic jargon.
Traditional text-prompting chatbots fail these users. Sahaayika flips the paradigm to Proactive Visual & Audio Guidance:
- Speaks First in Vernacular: Immediately greets and guides in native regional languages (Hindi, Tamil, Bengali, Telugu, Marathi, English).
- Visual Spotlight Aperture: Ingests browser screens and uses Google Gemini 3.8 Flash to physically highlight exact input fields with a glowing gold spotlight on top of the live web browser.
- Empathetic Companion Persona ("Didi"): Answers citizen doubts in simple, encouraging terms without confusing technical jargon.
The Sahaayika Brain backend is deployed live on Render:
- Base URL:
https://sahaayika-brain.onrender.com - API Documentation (Swagger UI):
https://sahaayika-brain.onrender.com/docs - System Health Check:
https://sahaayika-brain.onrender.com/health - Interactive Welfare Simulation Portal:
https://sahaayika-brain.onrender.com/portal
The APK size has been reduced by 81% (from 16.3 MB down to ~3.09 MB) using full R8 dead-code stripping, resource shrinking, and dual CPU architecture specialization:
- 📱 Download for Physical Phones (ARM64-v8a) (~3.09 MB)
Optimized for modern Android smartphones (MediaTek, Qualcomm Snapdragon, Samsung Exynos). - 💻 Download for Emulators (x86_64) (~3.09 MB)
Optimized for Android Studio desktop emulators running on PC/Mac.
- Accessibility Service (
SahaayikaAccessibilityService): Enables passive screenshot capture when navigating scheme portals to compute grounding coordinates without breaking user flow. - Display Over Other Apps (
TYPE_APPLICATION_OVERLAY): Renders the floating "Didi" avatar and the dynamic visual spotlight cutout over web browsers. - Microphone (
RECORD_AUDIO): Enables vernacular voice search and intent resolution.
┌─────────────────────────────────────────────────────────────────┐
│ Android Thin Client (Edge / Kotlin) │
│ │
│ [LanguageActivity] ──> [GuidedHubActivity] ──> [Scheme Portal] │
│ │ │ │ │
│ (Vernacular TTS) (Voice Intent Mic) (Chrome / Web) │
│ │ │
│ ┌───────────────────────┐ ┌───────────────▼──────┐ │
│ │ FloatingDidiAvatar │ │ SahaayikaA11yService │ │
│ │ (Draggable & Speech) │ │ (Silent Snapshot) │ │
│ └───────────▲───────────┘ └───────────────┬──────┘ │
│ │ │ │
│ ┌───────────┴───────────┐ │ │
│ │ SpotlightCanvasView │◄────────[box_2d & TTS]──────┤ │
│ │ (Clear Aperture + Ring)│ │ │
│ └───────────────────────┘ │ │
└────────────────────────────────────────────────────────┼────────┘
│
HTTPS / REST API │ (Multipart / JSON)
▼
┌─────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (Cloud / Render) │
│ │
│ • POST /api/v1/intent ──> Scheme matching engine │
│ • POST /api/v1/analyze-screen ──> Gemini 3.8 Flash Vision │
│ • POST /api/v1/didi/ask ──> Empathetic Conversational QA │
│ • GET /api/v1/schemes ──> Multilingual catalog │
│ │
│ ┌────────────────────┐ │
│ │ Google Gemini API │ │
│ │ (gemini-3.8-flash) │ │
│ └────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
- Uses high-speed screen understanding with normalized
box_2dcoordinates:[ymin, xmin, ymax, xmax]. - The Android client translates normalized coordinates into device physical pixels via
SpotlightCanvasView. - Clears a transparent portal aperture using
PorterDuff.Mode.CLEARand animates a pulsing 24K gold border around the active field.
- Citizens simply tap the glowing microphone button and speak freely in their native dialect (e.g., "मेरी लाडली बहना के पैसे देखने हैं").
- The intent pipeline resolves the goal, suggests matching official schemes, and automatically launches the official government portal.
- Integrates
IndicTextToSpeechwith regional locales (hi-IN,mr-IN,ta-IN,te-IN,bn-IN,en-IN). - Automatically narrates every step and guides first-time users before they even need to ask.
- Passively monitors UI element types to mask sensitive password or financial inputs before screenshot dispatch.
- Debounced accessibility event queue (1500ms) with concurrency mutex prevents duplicate network calls.
- Python 3.11+
- Android Studio / Android SDK 34 (Java 17+)
- Docker & Docker Compose (Optional)
- Google Gemini API Key (Get one here)
-
Navigate to the backend directory:
cd backend -
Create and activate a virtual environment:
python -m venv .venv # Windows: .venv\Scripts\activate # macOS/Linux: source .venv/bin/activate
-
Install dependencies:
pip install -r requirements.txt
-
Configure environment variables: Create a
.envfile inbackend/:GEMINI_API_KEY=your_gemini_api_key_here GEMINI_MODEL=gemini-3.8-flash DEFAULT_LANGUAGE=hi-IN PORT=8000
-
Start the server:
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
-
Run Backend Test Suite:
pytest -v # 71 passed in ~5.5s
To run the containerized backend locally:
docker compose up -d --buildHealth check verification:
curl http://localhost:8000/health
# {"status":"ok","service":"sahaayika-brain","version":"1.0.0"}- Open the
androidfolder in Android Studio. - Sync Gradle dependencies.
- Verify or update backend URL in
SahaayikaApiService.kt:// For live cloud: const val DEFAULT_BASE_URL = "https://sahaayika-brain.onrender.com/api/v1" // For local emulator: // const val DEFAULT_BASE_URL = "http://10.0.2.2:8000/api/v1"
- Build and run unit tests:
cd android ./gradlew testDebugUnitTest # 27 tests passed
- Build the debug APK:
./gradlew assembleDebug # Output: android/app/build/outputs/apk/debug/app-debug.apk
| Component | Test Suite | Tests | Result |
|---|---|---|---|
| Backend API | pytest tests/ |
71 Tests | ✅ Passing |
| Android Client | testDebugUnitTest |
27 Tests | ✅ Passing |
| E2E Visual Grounding | Live Android 14 Emulator | Tested | ✅ Aperture Drawn |
| Audio Vocalization | IndicTextToSpeech |
Tested | ✅ Vernacular Audio |
This project is licensed under the MIT License.