Outrageous Voice Assistant

The Outrageous Voice Assistant is a fully local voice assistant demonstration designed with an intuitive FastAPI backend and a straightforward HTML front-end. This project leverages open-weight models for Automatic Speech Recognition (ASR), Language Model (LLM), and Text-to-Speech (TTS) capabilities—all while running locally. No data is transmitted to the Internet, ensuring privacy and security.

Key Features

• Fully Local Operation: Enjoy the benefits of a voice assistant without compromising data privacy. The entire setup operates locally on affordable hardware.
• Speedy Performance: On systems with an RTX 5070 GPU and 12GiB VRAM, the round-trip audio processing time is around one second using the Kokoro TTS model.
• Voice Cloning: The project includes support for voice cloning using simple audio clips with corresponding transcriptions, allowing for personalized interaction.

How It Works

1. The frontend captures audio input from the user and sends the data to the backend via the /chat endpoint.
2. The backend processes the audio, extracting the sample rate and channel information.
3. The audio is transcribed to text using the ASR model.
4. The transcribed text is processed by the LLM, generating a response.
5. The response is converted to speech using a TTS model, normalized, and the audio is sent back to the frontend for playback.

Models Utilized

• ASR: NVIDIA parakeet-tdt-0.6b-v3 600M
• LLM: Mistral ministral-3 3b 4-bit quantized
• TTS (Simple): Hexgrad Kokoro 82M
• TTS (With Voice Cloning): Qwen3-TTS

Demos

• Voice assistant using Dua Lipa's voice clone: Demo Video
• Voice assistant using the default voice: Demo Video

Future Improvements

• Add support for Apple Silicon (MLX).
• Implement client-side Voice Activity Detection (VAD).
• Enhance voice cloning capabilities to eliminate the need for prepared transcripts.
• Introduce orchestration for more complex task detection and tool calls.

Disclaimer & Ethical Considerations

This project serves as a proof-of-concept and is provided "as is" without warrants. It is intended for educational and experimental use only. Voice cloning is meant for learning, and users are encouraged to seek permissions for real-life applications. The project also raises important ethical queries about the potential misuse of voice cloning technology, which can be achieved easily and quickly with minimal audio input, stressing the importance of responsible use.