Minnarone Framework

The Minnarone Framework offers a robust solution for developing AI agents capable of perceiving live multimodal contexts—including audio, video, chat, and platform events—and responding in real time. This framework can be employed for various interactions, serving as both a public participant (e.g., a co-host on a stream) and a private assistant (e.g., providing suggestions to sellers or meeting participants).

Originating from the successful implementation of an AI bot that engaged with live Twitch streams, this framework preserves the core principles of perception and reaction, ensuring it operates unnoticed by users, akin to passing a Turing test in chat environments.

Features

• Live Multimodal Context Perception: Capable of processing audio, chat, and video feeds simultaneously, enabling informative responses based on rich context.
• Proactive Interaction: AI agents can participate in chat or provide suggestions without sending direct messages, ensuring compliance with conversational norms and privacy guidelines.

Quickstart: Chat-only Shadow Rehearsal

Users can initiate a chat-only shadow rehearsal where Minnarone observes the chat, constructs possible responses, and stores evidence locally without posting messages. The following commands outline the steps for setting up this feature:

git clone https://github.com/carlitose/minnarone.git
cd minnarone
uv sync --extra tui
cp .env.example .env
mkdir -p .local/minnarone/onboarding/examplechannel
cp -R examples/onboarding/. .local/minnarone/onboarding/examplechannel/

Make necessary adjustments in the local configuration files, set up bot credentials, validate, and run the following commands:

uv run python -m minnarone .local/minnarone/onboarding/examplechannel/twitch-chat-shadow.it.yaml --check
uv run python -m minnarone .local/minnarone/onboarding/examplechannel/twitch-chat-shadow.it.yaml --tui

System Management

Minnarone employs a structured configuration system to manage different aspects:

• YAML Configuration: Use YAML files to select adapters, modes, and models, defining operational parameters.
• Soul and Facts: Define identity and stable facts in soul.md and facts/ directories.
• Prompt Templates: Enhance runtime instructions using customizable prompt templates located in the prompts_dir.

Progressive Implementation Paths

The framework provides several progressive paths for implementation:

1. Chat-only shadow (Initial setup).
2. Media components (testing audio and video separately).
3. Full multimodal setup (advanced configuration for diverse hardware profiles).
4. Live operation (requires consent and specific configurations).

Safety Mechanisms

To ensure responsible operation, the framework includes safety gates:

• Public Send Protocols: Default settings prioritize safety with alert budgets to prevent excessive messaging.
• Allow-Listing Channels: Live interactions require channels to be on an allow-list, ensuring compliance with community guidelines.
• Token Verification: Regular validation of authentication tokens at runtime to maintain security and prevent unauthorized actions.

Additional Capabilities

Minnarone supports flexible deployment scenarios, such as acting as a local commentator during meetings or events. The system captures audio and video streams, providing insights without external messaging capabilities, making it suitable for various professional applications.

For comprehensive documentation and advanced features, refer to the project specification, Twitch operator guide, and meeting assistant guide. This innovative AI framework is designed to adapt fluidly to user requirements while maintaining ethical usage through stringent safety measures.