Quartz is a powerful on-device inference engine designed for high-performance Natural Language Processing (NLP) and image generation. Built from scratch in Rust, this engine eliminates the need for cloud services, enabling seamless local execution of language model (LLM) chat and image generation tasks directly on mobile and desktop devices.

Key Features

• Complete Local Processing: With Quartz, both chat and image generation are performed entirely on-device, ensuring user privacy and reducing latency. The engine supports advanced capabilities, such as generating full text-to-video clips, exemplified by Saient mobile’s integration with the Wan2.1 T2V video model.

• Dependency-Light Architecture: Designed for efficiency, Quartz operates with a minimal, mostly unsafe-free codebase of approximately 15,000 lines, streamlining the experience and focusing on performance across various platforms, including stock Android devices and common desktop setups.

Capabilities

Quartz combines two primary functionalities in a single binary:

1. LLM Chat Inference: Involves loading GGUF models, tokenization using native GGUF vocab or Jinja2 templates, executing the forward pass, and offering an OpenAI-compatible HTTP server for streaming completions.
2. Image Generation: Implements a custom pipeline for Stable Diffusion (SD1.5 and SDXL), utilizing a unique CLIP text encoder, UNet, VAE, and scheduling mechanisms driven by simple command-line options.

Flexible Backends

• CPU Support: Runs efficiently on standard CPU architectures with optimized AVX2/FMA paths to enhance performance.
• CUDA and Vulkan: Supports specialized GEMV kernels for CUDA-enabled GPUs and Vulkan for high-end graphics processing, making it suitable for various computing environments, including Android devices.

Quantization and Format Support

Quartz directly reads GGUF files and supports multiple quantization formats, including Q4_K, Q6_K, Q8_0, Q5_0/Q5_1, and more, providing flexibility in model usage.

API Accessibility

The engine includes an HTTP API compatible with OpenAI, facilitating easy integration with existing applications. The API provides endpoints such as:

GET  /health
GET  /v1/models
POST /v1/chat/completions (SSE streaming)

This allows both desktop and mobile applications to communicate with the engine using the same protocol.

Project History

Originally developed under the name tinyq4, the project has since rebranded to Quartz, reflecting its evolution and development focus. The renaming provides a clearer association with its robust capabilities in the context of modern AI applications.

Explore the Quartz engine to empower on-device machine learning tasks with privacy and efficiency that adapts to the needs of contemporary computing environments.