Submitted under [Rapid Crisis Response] Open Innovation track — Google Solution Challenge 2026, hosted on Hack2Skill.
Problem: Hospitality venues face life-threatening emergencies where language barriers and fragmented communication between panicked guests, staff, and first responders create dangerous delays. No existing tool detects emotional distress in real time across languages.
Solution: ToneLens is a real-time emotional intelligence agent that detects panic, urgency, and distress in spoken communication across any language — surfacing instant translation, emotion readings, urgency levels, and staff response suggestions in under 2 seconds. Deployed on Google Cloud Run. No app install required.
Google Technologies Used:
- Gemini Live API (gemini-2.5-flash-native-audio-latest) — real-time multimodal streaming
- Vertex AI (gemini-2.0-flash) — structured output reformatting
- Google Cloud Run — serverless deployment
- Google Cloud Firestore — session memory
- Google ADK — agent framework
Google Translate tells you the words. ToneLens tells you the truth.
ToneLens is a real-time emotional intelligence web app built for the Google Solution Challenge 2026. It watches a conversation through the browser camera, listens to live audio, streams both to Gemini Live, reformats the model output into a strict UI-friendly structure, and surfaces translation, emotion, subtext, tactical suggestions, agent actions, history, and audio playback in one interface.
Live URL: https://tonelens.mohanprasath.dev
The current app implements:
- FastAPI routes for
/,/app,/about,/health, and/api/history/{session_id}. - A WebSocket endpoint at
/ws/{session_id}for live camera, microphone, mode, and location streaming. - A Gemini Live bridge using
gemini-2.5-flash-native-audio-latestwithresponse_modalities=["AUDIO"]. - A second-pass formatter using Vertex AI
gemini-2.0-flashto coerce live output into strict labeled lines. - Six user modes: travel, meeting, present, negotiate, crisis_response, hospitality.
- Keyword-driven agent actions for cultural tips, emergency help, meeting-note capture, and stress reporting.
- Firestore-backed session history and meeting notes.
- A vanilla HTML/CSS/JS frontend with live camera, live microphone, stress graph, history sidebar, meeting-notes sidebar, negotiation power balance, score ring, whisper coach, transcript feed, and real-time audio playback.
From backend/main.py:
GET /servesfrontend/landing.htmlGET /appservesfrontend/index.htmlGET /aboutservesfrontend/about.htmlGET /healthreturns a JSON health payloadGET /api/history/{session_id}returns the last 10 exchanges for a Firestore sessionWS /ws/{session_id}handles realtime frontend traffic
graph TD
A[Browser UI] -->|JPEG frames| B[FastAPI websocket]
A -->|16kHz PCM audio as base64| B
A -->|mode + location| B
B --> C[GeminiBridge]
C -->|video=image/jpeg| D[Gemini Live\ngemini-2.5-flash-native-audio-latest]
C -->|audio=audio/pcm;rate=16000| D
D -->|audio chunks| C
D -->|output_transcription + model_turn text| C
C -->|reformat prompt| E[Vertex AI\ngemini-2.0-flash]
E --> C
C -->|structured UI events| A
C -->|log exchanges| F[Cloud Firestore]
C -->|keyword actions| G[backend.agent helpers]
G -->|cultural tips / emergency links / notes / stress| A
What the running code does:
- The frontend opens
WS /ws/{session_id}. - The backend creates a Firestore session and initializes
GeminiBridge. - Audio is queued in
main.pyuntil the live session is ready, then drained to Gemini. - Frames are throttled to one every 2 seconds and sent as JPEG blobs.
- Audio is base64-decoded server-side and sent as raw PCM bytes with
mime_type="audio/pcm;rate=16000". - Gemini Live returns audio plus transcript text via
server_content.output_transcriptionand/ormodel_turn.parts. - The bridge accumulates transcript text until
turn_complete. - The raw text is reformatted through Vertex AI
gemini-2.0-flash. - The structured response is parsed into translation, emotion, subtext, suggestion, and negotiation power.
- Exchanges are stored in Firestore and keyword actions may trigger extra UI panels.
The live prompts in backend/gemini_bridge.py define six actual operating modes:
travel: translation, emotion, subtext, suggestionmeeting: same four-line structure, plus meeting-note capture and screen-audio workflow in the frontendpresent: filler-word detection, pace analysis, and coaching tipsnegotiate: translation, emotion, tactical subtext, power score, and negotiation suggestioncrisis_response: detects panic, fear, urgency in speech. Flags emergency keywords. Rates urgency LOW/MEDIUM/HIGH/CRITICAL. Suggests de-escalation responses for staff.hospitality: detects guest frustration and confusion in any language. Suggests warm, culturally-aware, service-oriented responses.
The runtime helper functions in backend/agent.py are:
search_cultural_context(language, phrase)find_emergency_services(situation, latitude, longitude)save_meeting_note(session_id, key_point, speaker_emotion)get_stress_report(session_id)
Important implementation detail: the bridge explicitly notes that Gemini Live function declarations are omitted for this model/config, so these helpers are triggered by keyword logic in GeminiBridge._run_keyword_actions, not by live tool-calling.
From frontend/index.html, the app currently includes:
- Home link and mode switcher
- Connection status dot and label
- Live camera panel with emotion badge overlay
- Camera-denied error state with
[ RETRY ] - Emotion meter
- Stress tracker chart
- Live microphone waveform and status
- Translation card
- Cultural tip card
- Emotional context card
- Suggestion card
- Negotiation-only power balance, score ring, whisper coach, and momentum chart
- Agent actions card
- Transcript card
- History sidebar
- Meeting notes sidebar with export
- Emergency overlay with hospital, police, and
Call 112
From the code and dependency files:
- Backend: FastAPI, Uvicorn, Python
- AI SDK:
google-genai - Realtime transport: WebSockets
- Persistence:
google-cloud-firestore - Environment loading:
python-dotenv - Frontend: vanilla HTML, CSS, JavaScript
- Charts: Chart.js
- Container: Docker
- Deployment target: Google Cloud Run
- Image registry path in deploy script: Artifact Registry in
us-central1
Python dependencies from backend/requirements.txt:
fastapiuvicorn[standard]google-genaigoogle-adkgoogle-cloud-firestorepython-dotenvwebsocketspython-multipartaiofilespydantic
backend/
agent.py
gemini_bridge.py
main.py
session_manager.py
requirements.txt
frontend/
landing.html
about.html
index.html
Dockerfile
deploy.sh
README.md
- Python installed locally
gcloudinstalled and authenticated- Application Default Credentials for Firestore and Vertex AI access
- A valid
GOOGLE_API_KEYfor Gemini Live
The code reads these values directly:
GOOGLE_API_KEY: required for the Gemini Live session clientGOOGLE_CLOUD_PROJECT: used by the bridge and Vertex AI/Firestore integrationsGOOGLE_CLOUD_REGION: used by deployment and Vertex AI client setupPORT: used by the container runtime, default8080
Example .env:
GOOGLE_API_KEY=your_google_ai_studio_key
GOOGLE_CLOUD_PROJECT=notional-cirrus-458606-e0
GOOGLE_CLOUD_REGION=us-central1python -m venv .venv
. .venv/bin/activate
pip install -r backend/requirements.txt
gcloud auth application-default login
uvicorn backend.main:app --reload --port 8080Open:
http://localhost:8080
There are two deployment paths in the repo.
The project has been deployed with:
gcloud run deploy tonelens --source . --region us-central1 --project notional-cirrus-458606-e0deploy.sh does the following:
- Sets project
notional-cirrus-458606-e0 - Ensures Artifact Registry repo
tonelens-repoexists - Builds
us-central1-docker.pkg.dev/notional-cirrus-458606-e0/tonelens-repo/app:latest - Pushes the image
- Deploys to Cloud Run with:
- region
us-central1 - memory
2Gi - cpu
2 - min instances
0 - max instances
5 - port
8080
- region
Important: deploy.sh sets GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION, but it does not inject GOOGLE_API_KEY. For a fresh deployment, that key must already exist in the service config or be supplied separately.
From Dockerfile:
- Base image:
python:3.11-slim - Working directory:
/app - Installs
backend/requirements.txt - Copies
backend/andfrontend/ - Exposes port
8080 - Runs:
uvicorn backend.main:app --host 0.0.0.0 --port 8080 --workers 1 --log-level infoFrom backend/session_manager.py:
- New sessions are created in the
sessionscollection - Each session stores:
created_atmodeexchanges
- Exchanges append:
timestamptranslationemotionsubtextsuggestion
Meeting notes are stored under:
sessions/{session_id}/notes/{note_id}
From the app content itself:
- Name: Mohan Prasath
- Age: 18
- Location: Chennai, India
- Education: B.Tech CSE (AI & ML)
- Portfolio: https://mohanprasath.dev
ToneLens is not a static demo site. The current codebase already ships a real multimodal loop:
- browser camera and mic capture
- live WebSocket transport
- Gemini Live audio-only response flow
- structured second-pass reasoning format
- Firestore session memory
- emergency and cultural-action side effects
- mobile-optimized landing, about, and app pages
- Six purpose-built AI modes, each with a distinct Gemini system prompt tuned for a specific context
- Crisis Response mode flags urgency level (LOW/MEDIUM/HIGH/CRITICAL) in real time
- Hospitality mode detects guest frustration across any language with culturally-aware suggestions
- Camera is fully optional — app works via voice only, camera panel hides gracefully if denied
- Mobile responsive — works on any device, no installation
This repo is already in a shape where a judge can open the landing page, launch the app, grant camera/mic access, and see a working multimodal interaction pipeline.
See LICENSE.