bytesized

bytesized: A Compact and Efficient C Engine for SmolLM2-135M

Overview
The bytesized project features a 2,946-byte C program capable of running SmolLM2-135M, an actual language model (LLM) that outputs tokens precisely matching those generated by Hugging Face transformers. The program directly utilizes the model.safetensors and tokenizer.json files without requiring any conversion. Its compact size allows it to fit within a single QR code, showcasing the potential of minimalistic programming.

The entire program encapsulated as a QR code.

Getting Started
To retrieve and execute the engine, make sure to follow the instructions below. Note that smol, an optional Python front-end, is available for running the binary and rendering the output, though it is not included within the minimal engine size:

zbarimg --raw --oneshot -Sbinary qr.png > final && chmod +x final

Usage
Running the engine is straightforward. The command format is as follows:

/path/to/bin/final "prompt" [n_tokens]

For example, using the prompt "The cat sat":

/path/to/final "The cat sat" 50

Below is a sample output:

$ ./final "The cat sat" 50
╭──────────────────────────────────────────────────────────╮
│ ✻ smol                                                   │
│                                                          │
│ engine  2,946 bytes of C                                 │
│ model   SmolLM2-135M, unmodified safetensors             │
│ decode  greedy, matches HF transformers token for token  │
╰──────────────────────────────────────────────────────────╯

Technical Specifications

• The engine is designed specifically for Linux arm64 architectures and has been tested in isolated environments such as OrbStack on M4 Macs. It may also function effectively on devices like the Raspberry Pi 4/5 or AWS Graviton, though further validation on these systems hasn't been conducted.
• The program is built around several strict rules, ensuring it offers correctness in greedy decoding and maintains fidelity with top-tier models in the industry.
• It operates without libc, relying instead on five raw system calls to deliver optimal performance.

Technical Process
The development process involved rigorous optimization, systematically reducing size while maintaining functionality:

• By leveraging mmap for safetensors and employing byte-level BPE for tokenization, the engine achieves efficient reading and processing.
• The architecture combines secure handling of tensors with a streamlined transformer format, minimizing overhead while maximizing throughput.
• Detailed benchmarking showcases the performance metrics of the engine with respect to standard model executions, illustrating the trade-offs between size and speed.

Future Developments
Potential avenues for enhancement include:

• Further miniaturization to incorporate the unpacker within ELF header fields to exploit overlooked kernel functionalities.
• Exploring a context-mixing model that would provide new methodologies for token generation.
• Expanding compatibility to support x86-64 architectures, broadening the engine's accessibility.

Conclusion
The bytesized engine exemplifies the convergence of compact programming and powerful AI capabilities, providing users a remarkable tool for running LLMs with minimal resources while ensuring correctness and efficiency.