Voice Message Support for WhatsApp

[robertocarlous]

Problem Statement

Users can currently only type commands. Voice notes are a first-class WhatsApp media type, and speaking "deposit fifty dollars" is a meaningfully lower-friction interaction than typing it, especially for less text-comfortable users. This is a distinct feature from the existing text parser — it requires audio handling, transcription, and a materially higher tolerance for misrecognition on financial actions.

Current State

src/whatsapp/handler.ts currently processes text messages via Twilio's inbound webhook payload. There is no audio/media handling today. Intent parsing (src/nlp/parser.ts) is a closed union ('deposit' | 'withdraw' | 'balance' | 'earnings' | 'help' | 'unknown') resolved via a regex fast-path with a Claude Haiku fallback for anything the regex doesn't confidently match. Voice should feed into this exact same function — the value of this feature is entirely in getting from "audio" to "the same text a typed message would have produced," not in reimplementing intent handling for a new modality.

Proposed Solution

Integration Points

• Extend src/whatsapp/handler.ts to detect incoming messages with audio media (Twilio's MediaUrl/MediaContentType fields on voice notes) and download the audio via the authenticated Twilio media URL.
• Add a speech-to-text step behind an explicit provider interface (e.g. TranscriptionProvider with a single transcribe(audioBuffer): Promise<{ text: string; confidence: number }> method) so the actual STT vendor is swappable and not hard-coded into the handler — mirrors the existing pattern of abstracting the Stellar RPC client behind ResilientRpcClient rather than calling a specific provider's SDK directly from route handlers.
• Transcribed text is passed into the existing parseIntent(message) in src/nlp/parser.ts unchanged — no new intent types, no parallel voice-specific parsing logic.
• Confirmation-before-execution: any parsed intent that would move funds or change strategy, when it originated from voice, must be echoed back to the user ("I heard: withdraw $50 — confirm?") and require an explicit affirmative reply before executing. This is a materially different UX requirement from typed commands (which are already an explicit, deliberate action) and should be implemented as an explicit flag threaded through from the voice entry point, not inferred implicitly.
• Low-confidence transcriptions (below a configurable threshold from the provider's confidence score) should produce a "please repeat or type your command" response rather than attempting to parse and guess.

Edge Cases & Failure Modes

• Transcription succeeds but produces nonsensical/unparseable text — falls through to the existing 'unknown' intent handling, same as a nonsensical typed message would.
• Transcription provider is down/errors — respond with a clear "voice messages aren't available right now, please type your command" rather than a silent failure or hang.
• Unsupported audio format/codec from a given device — detect and respond gracefully rather than passing an unsupported buffer into the transcription provider and erroring opaquely.
• User sends a voice note that says "yes" as the confirmation reply to a pending voice-originated action — confirmation replies should be accepted via either voice or text, since it would be inconsistent to require typing to confirm a spoken command.

Security & Privacy Considerations

• Audio content is more sensitive than text in some jurisdictions (biometric/voice-data regulations) — document what is and isn't retained (e.g. is the raw audio buffer discarded immediately after transcription, or stored?) and default to not persisting raw audio unless there's a specific reason to.
• The confirmation-before-execution requirement is itself a security control against misrecognition-driven unintended transactions — it must not be skippable or bypassable for any financial intent, and this should be covered by a specific test asserting a voice-originated deposit/withdraw intent never executes without a prior confirmation round-trip.

Out of Scope

• Real-time/streaming transcription — process complete voice notes only (matches how WhatsApp delivers them).
• Voice output (the agent replying with synthesized speech) — text/WhatsApp-formatted replies only, even for voice-originated requests.

Suggested Implementation Plan

1. TranscriptionProvider interface and one concrete implementation; audio detection/download in handler.ts.
2. Wire transcribed text into the existing parseIntent pipeline; low-confidence fallback response.
3. Confirmation-before-execution flow for financial intents originating from voice.
4. Fixture-based tests (clear command, ambiguous command, unsupported audio, provider failure).

Acceptance Criteria

• Incoming voice notes are downloaded, transcribed, and routed through the existing parseIntent function with no new parsing logic
• Any deposit/withdraw/strategy-change intent originating from voice requires an explicit confirmation round-trip before executing — verified by a test that a voice-originated financial intent never executes on the first pass
• Low-confidence transcriptions produce a "please repeat or type" response instead of an attempted parse
• Transcription provider is abstracted behind a documented, swappable interface
• Provider outage produces a clear user-facing fallback message, not a silent failure
• Tests using fixture audio covering: a clear command, an ambiguous/low-confidence command, unsupported audio format, and provider failure
• Documentation of what audio data is retained vs. discarded after transcription

[Anuoluwapo25]

Hi maintainer, I’d love to work on this issue. I’m comfortable implementing WhatsApp voice note handling.

[grantfox-oss]

🦊 GrantFox — @Anuoluwapo25 has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #288)
2. Your PR will be reviewed by the Neurowealth maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign | FWC26 campaign!

@Anuoluwapo25's PR #299 was approved and merged by @robertocarlous.

🏆 @Anuoluwapo25: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (627 total points). Track your full progress on GrantFox.

👏 Great work, @Anuoluwapo25! Keep contributing to Neurowealth.