llmedge is a lightweight and efficient Android library designed for running GGUF language models directly on-device, leveraging the power of llama.cpp. This library enables various AI functionalities including language inference, speech transcription, text-to-speech synthesis, and image generation, all while ensuring enhanced performance and memory management suitable for mobile devices.

Key Features

• LLM Inference: Seamlessly run GGUF models on Android with llama.cpp integration.
• Model Management: Facilitate easy downloading and caching of models from Hugging Face Hub, ensuring quick access to AI models.
• Optimized Conversational AI: Supports KV Cache reuse for efficient multi-turn conversations.
• Speech-to-Text (STT): Integration with Whisper.cpp, offering capabilities like streaming transcription, language detection, and subtitle generation in SRT format.
• Text-to-Speech (TTS): Utilizes Bark.cpp with ARM optimizations, enabling high-quality speech synthesis.
• Image Generation: Generate images using Stable Diffusion with integrated EasyCache and LoRA support for diverse applications.
• Video Generation: Create short videos via Wan 2.1 models, handling the sequential loading of frames effectively.
• On-device Retrieval-Augmented Generation (RAG): Implement features such as PDF indexing, embeddings, vector search, and Q&A directly on the device.
• Optical Character Recognition (OCR): Utilize Google ML Kit for efficient text extraction from images.
• Memory Monitoring: Built-in RAM usage tracking to ensure optimal performance.
• Vision Models Compatibility: Prepared to accommodate LLaVA-style models, demanding specific model formats.
• Vulkan Acceleration: Optional GPU acceleration is available for devices running Android 11+ that support Vulkan 1.2.

Getting Started

The library can be utilized through the LLMEdgeManager singleton, which manages model loading, threading, and caching operations. Here’s how to generate text using a pre-trained model:

CoroutineScope(Dispatchers.IO).launch {
    val reply = LLMEdgeManager.generateText(
        context = context,
        params = LLMEdgeManager.TextGenerationParams(
            prompt = "Summarize on-device LLMs in one sentence."
        )
    )
    withContext(Dispatchers.Main) {
        outputView.text = reply
    }
}

Advanced Functionalities

1. Speech-to-Text Transcription:

val text = LLMEdgeManager.transcribeAudioToText(
    context = context,
    audioSamples = audioSamples
)

2. Text-to-Speech Synthesis:

val audio = LLMEdgeManager.synthesizeSpeech(
    context = context,
    params = LLMEdgeManager.SpeechSynthesisParams(
        text = "Hello, world!"
    )
)

3. Image Analysis:

val description = LLMEdgeManager.analyzeImage(
    context = context,
    params = LLMEdgeManager.VisionAnalysisParams(
        image = bitmap,
        prompt = "Describe this image in detail."
    )
)

4. Video Generation:

val frames = LLMEdgeManager.generateVideo(
    context = context,
    params = VideoGenerationParams(
        prompt = "a cat walking in a garden, high quality",
        videoFrames = 8
    )
)

System Requirements

• Android NDK and CMake are required for building and utilizing Vulkan acceleration.
• Supports devices running Android 11 and above for optimal functionality and performance.

Early Development Notice

While powerful, llmedge is currently in the early stages of development and features may evolve significantly. For further insights and examples, refer to the examples repository where sample usages and implementation details are showcased.