PasLLM is an efficient LLM inference engine designed for local execution using pure Object Pascal. With support for multiple architectures and advanced quantization formats, it enables optimized model deployment without dependencies on other languages. Aimed at cross-platform compatibility, PasLLM combines performance with easy accessibility for developers.
PasLLM is a high-performance Large Language Model (LLM) inference engine developed entirely in pure Object Pascal. This implementation enables users to run LLMs locally with a range of optimized quantization and inference features, making it a powerful tool for developers working with machine learning models.
PasLLM features multiple quantization formats aimed at achieving an excellent balance of model quality and size:
PasLLM includes pre-quantized models which can be found at Mega.nz. Supported models include:
# Execute inference with a quantized model
./bin/pasllmcli -model=bin/models/qwen2.5_0.5b_instruct_q40nl.safetensors
For those who wish to build from source, the following commands are available: FreePascal:
fpc -O3 src/pasllmcli/pasllmcli.dpr
Delphi: Open src/pasllmcli/pasllmcli.dproj in the Delphi IDE and initiate the build.
PasLLM is structured to ensure that the core inference engine and applications are easily navigable and modifiable. It supports compiling on various platforms without relying on third-party dependencies, allowing developers to focus on their projects effectively.
Models from platforms such as Hugging Face can be seamlessly converted for use in PasLLM with provided scripts, enabling unprecedented flexibility in model utilization for developers.
Comprehensive documentation is available for the quantization formats and deeper insights into the functionalities of PasLLM, ensuring users have all the necessary tools for successful deployment of LLMs.
No comments yet.
Sign in to be the first to comment.