PulsarForge

PulsarForge is a powerful CPU-only LLM inference engine developed in C11, designed to efficiently manage a 744-billion-parameter model on standard hardware such as a 2018 laptop with 32GB of RAM and a USB SSD. This project encompasses five distinct architectures, each validated for bit-exactness against an independent oracle, ensuring robust performance on both Windows and Linux platforms. A comprehensive engineering retrospective, including metrics from various attempts, provides transparency in its development journey.

Current Status: Research Engine
Windows is the primary and most extensively measured environment, with Linux (native or WSL) validated as a secondary target. For a first-time setup, refer to GETTING_STARTED.md to build the system, bootstrap the tokenizer, perform a one-time conversion, and start using the chat and API interface in five straightforward steps.

PulsarForge initially began as a didactic tool for the Qwen3-0.6B model, enabling users to grasp inference mechanisms through hands-on experience and validation against the HuggingFace reference. It has since evolved into a unique offering in the landscape of machine learning, showcasing remarkable features:

• Architectural Validation: Five architectures (qwen3, qwen3moe, deepseek2, olmoe, and glm-dsa), each meticulously validated for bit-exact or argmax-exact results against a reliable oracle.
• Expert Streaming: Achieving expert streaming on a pure CPU setup, where a 202GB GGUF file is stored on a USB SSD. Experts are dynamically loaded per token using a NO_BUFFERING reader with an LRU cache, ensuring optimal performance with high recall rates and efficient compute/IO overlaps.
• Rigorous Measurement Discipline: A commitment to transparency is evident in every measurement shared, detailing experimental setups and findings, including failures, as documented in the RETROSPECTIVE.md.

Performance Insights

On a HP EliteBook 840 G5 (equipped with an i7-8550U processor, 32GB RAM, and a USB SSD), PulsarForge has demonstrated impressive performance metrics:

Novel Findings

Research has yielded several novel findings, including:

• Expert Relevance: Strong correlations between router weights and true expert importance, challenging previously held assumptions about model optimization.
• Model Density: Evidence that the GLM-5.2 model exhibits higher density, negating the need for optimizations focused on deduplication and part-skipping due to lack of redundant experts.
• Unique Context: The project occupies a unique niche of operating a 700B-class model with considerably lower resource requirements than many competitors.

Integration and Usability

PulsarForge includes a custom .forge container format designed to optimize the performance and manageability of models. This container format was developed based on A/B testing against other formats, ensuring reliability and efficiency during model operations.

Usage examples for daily operations include:

python tools/forge_convert.py --dest glm52.forge
python tools/forge_idx.py glm52.forge
nf chat glm52.forge --session my.nfs

Additional details regarding ecosystem comparisons, historical context, and runtime commands can be found in the documentation: OPERATIONS.md for English and README.it.md for original Italian documentation.

Key Takeaway

PulsarForge exemplifies cutting-edge development in large language model inference techniques, leveraging standard hardware to achieve unprecedented performance metrics in the 700B-plus model realm. This project serves as a valuable resource for researchers and developers seeking to understand the complexities and efficiencies of modern machine learning frameworks.