hermes-live2d

ATRI Live2D: Your Interactive Desktop Pet

ATRI Live2D combines the magic of digital art with advanced interactive technology to create a unique transparent desktop pet, powered by Tauri 2 and pixi-live2d-display. This application features a charming character named Sparkle (火花), who can engage and entertain users with voice lip sync, dynamic expressions, mouse tracking, and a local HTTP API.

Key Features:

• Interactive Desktop Presence: Enjoy a frameless, transparent, always-on-top desktop pet that brings life to your workspace.
• Real-Time Mouse Tracking: Sparkle’s eyes and head follow your cursor across multiple displays, creating an engaging interactive experience.
• Voice Lip Sync: The character's mouth movements are synchronized with audio playback, providing a lifelike interaction as it 'speaks' your text-to-speech (TTS) recordings.
• Dynamic Expressions: Choose from 12 different expressions that can be triggered programmatically, allowing for a variety of interactions.
• Speech Bubbles: Communicate messages through customizable typewriter-style speech bubbles that emerge above Sparkle’s head.
• Local REST API: Utilize a local HTTP API for seamless integration with external programs and remote control. Access it at http://127.0.0.1:3210.
• Persistent Window State: The desktop pet remembers its window size and position across sessions, ensuring a consistent user experience.

How It Works

ATRI Live2D is designed to work with the hermes agent. When the agent generates a TTS recording, it can upload audio to Sparkle using the hermes-live2d-voice skill, leading to synchronized audio, lip movements, and expressions. To deploy this configuration:

1. Host the Hermes agent remotely (e.g., on EC2 or another server) and run ATRI Live2D on your Mac.
2. Install and configure the hermes-live2d-voice skill to communicate between your Hermes agent and Sparkle.
3. Ensure both systems are connected using Tailscale for secure, remote access without exposing the API.

Example API Usage

Here are some examples of how to control your desktop pet through the API:

# Health check
curl http://127.0.0.1:3210/status

# Make the pet speak (using audio and expression)
curl -X POST http://127.0.0.1:3210/speak \ 
  -H 'Content-Type: application/json' \ 
  -d '{"text": "hello!", "audio_url": "file:///path/to/audio.wav", "expression": 1}'

# Change the pet's expression (1-12)
curl -X POST http://127.0.0.1:3210/expression \ 
  -H 'Content-Type: application/json' \ 
  -d '{"name": "exp1"}'

# Display a speech bubble
curl -X POST http://127.0.0.1:3210/bubble \ 
  -H 'Content-Type: application/json' \ 
  -d '{"text": "thinking...", "duration": 3000}'

For more detailed API endpoint references, consult the API documentation.

Technical Stack

• Desktop Framework: Tauri 2 (Rust)
• Rendering Library: PixiJS 6 + pixi-live2d-display
• Live2D Technology: Cubism 4 SDK
• API Server: Axum

Limitations

• ATRI Live2D is currently supported only on macOS (Apple Silicon).
• Relies on macOS-specific APIs and features for its functionality, including transparency and mouse tracking.
• Audio-driven lip sync may not achieve phoneme accuracy; mouth movements are based on volume.

Conclusion

ATRI Live2D offers a creative and interactive solution for users seeking to enhance their desktop environment with animated characters. By leveraging TTS powered by a hermes agent, Sparkle transforms mundane tasks into engaging interactions, making work and leisure more enjoyable.