paddock

paddock is a command-line application designed to optimize the selection and launch of local large language models (LLMs) on Apple Silicon Macs. It answers essential questions regarding model compatibility, speed estimates, and execution commands, streamlining the user experience for machine learning practitioners.

Key Features

• Compatibility Assessment: Quickly identifies which LLMs fit based on macOS memory limitations, cross-referencing parameter counts and quantization formats with available RAM.
• Speed Estimation: Provides real-time performance predictions based on Apple Silicon's memory bandwidth variation across different models, ensuring users understand how fast a model will operate under specified conditions.
• Streamlined Execution: Users can launch models with a single command, bypassing complex ollama commands and detailed configurations required by various runtime environments.

Design Considerations

paddock is built with Apple Silicon in mind, optimizing for unified memory architecture without the need for a separate VRAM pool. It directly reads Metal's maximum working set limits to determine which models can fully utilize GPU resources. Performance estimates are refined using a table of bandwidth values tailored to each silicon chip, ensuring accurate predictions for models ranging from M1 to M5.

Usage Overview

When the paddock command is executed without parameters, it opens an interactive terminal user interface (TUI) that displays the model catalog, allowing users to navigate through available LLMs, check their details, and launch or serve selected models. Users can use specific commands to scan their machine's compatibility, refresh the model catalog, and receive ranked recommendations based on their use case.

Example Commands

• Scan for Compatibility:

  paddock scan
  

• Refresh Models Catalog:

  paddock sync
  

• Launch a Model:

  paddock run <model_name>
  

• Serve a Model (OpenAI-compatible endpoint):

  paddock serve <model_name>
  

Advanced Features

paddock supports detailed logs and server management functionalities, allowing users to stop, restart, and view logs for all running models. Additionally, it features a menu bar integration for quick access to active endpoints and model statuses. Each subcommand can also be executed in machine-readable JSON format, facilitating automated workflows.

Future Development

The roadmap for paddock includes plans for developing a desktop application, implementing real measured token generation speeds, and expanding compatibility to Linux and Windows environments.

By providing a comprehensive toolkit for managing and deploying LLMs exclusively on Apple Silicon, paddock aims to enhance the productivity of machine learning developers and researchers.