AssistantAI: A Real-Time Desktop AI Assistant

AssistantAI is a cutting-edge real-time desktop assistant designed to enhance productivity by utilizing advanced AI capabilities. This tool seamlessly integrates with selected applications such as Zoom, Teams, Telegram, and various browsers to provide an innovative way to handle spoken conversations.

Key Features

• Application-Specific Audio Capture: Leveraging ScreenCaptureKit to capture audio specific to any selected application.
• Voice Activity Detection: Employing Silero VAD for accurate detection of spoken audio.
• Local Speech Recognition: Utilizing whisper.cpp to convert spoken words into text locally with high accuracy.
• Context-Aware Conversations: The assistant uses attached documents to generate relevant responses, ensuring a smarter interaction experience.
• Screenshot Analysis: Capture and analyze screenshots for further context and assistance directly through the intuitive web interface.
• Local Conversation History: Keep track of interactions for easy reference and continuity in conversations.
• Remote Web Interface: A user-friendly interface to manage and interact with the assistant remotely.

Demo

For a visual demonstration of AssistantAI's capabilities, view the following video: Watch the video.

Development and Future Enhancements

Built as a personal engineering project, AssistantAI explores the realms of low-latency desktop AI, speech processing, and intelligent conversation management. The current implementation is crafted in Python, with plans for future enhancements including:

• Support for additional LLM backends like Ollama and llama.cpp.
• Compatibility with more document formats.
• Enhanced capabilities for screenshot analysis to improve accuracy and insight.

Useful Resources

• Python Setup: A guide to setting up the Python environment can be found in bash python/run/setup_python_env.sh.
• Documentation: Further details on Python implementation can be accessed in python/README.md.
• Audio Capture Research: Explore audio capture methodologies in docs/modules/audio-capture-research.md.

This project stands to evolve with continuous improvements and the introduction of new features, making AssistantAI a valuable asset for users seeking an intelligent and responsive desktop assistant.