create-llm

npx @theanikrtgiri/create-llm my-awesome-llm
cd my-awesome-llm
pip install -r requirements.txt
python training/train.py

Why choose create-llm?

Training a language model from scratch can be a daunting task, requiring knowledge of several components, including:

• ✅ Model architecture (like GPT, BERT, T5)
• ✅ Data preprocessing pipeline
• ✅ Tokenizer training
• ✅ Training loop with callbacks
• ✅ Checkpoint management
• ✅ Evaluation metrics
• ✅ Text generation capabilities
• ✅ Deployment tools

create-llm streamlines this by providing all necessary tools in one command.

Key Features

🎯 Right-Sized Templates

Choose from four templates optimized for various use cases:

• NANO (1M parameters) - Learn in 2 minutes on any laptop
• TINY (6M parameters) - Prototype in 15 minutes on CPU
• SMALL (100M parameters) - Production-grade models in hours
• BASE (1B parameters) - Research-ready models in days

🔧 Comprehensive Toolkit

Includes everything needed out of the box:

• PyTorch training infrastructure
• Data preprocessing pipeline
• Tokenizer training (BPE, WordPiece, Unigram)
• Checkpoint management with auto-save
• TensorBoard integration
• Live training dashboard
• Interactive chat interface
• Model comparison tools
• Deployment scripts

📊 Smart Defaults

Features intelligent configuration that:

• Auto-detects vocabulary size from tokenizer
• Warns about model/data size mismatches
• Detects overfitting during training
• Suggests optimal hyperparameters
• Handles cross-platform paths

🎨 Plugin System

Optional integrations include:

• WandB - Experiment tracking
• HuggingFace - Model sharing
• SynthexAI - Synthetic data generation

Quick Start

🚀 One-Command Setup

# Using npx (recommended - no installation needed)
npx @theanikrtgiri/create-llm my-llm

# Or install globally
npm install -g @theanikrtgiri/create-llm
create-llm my-llm

🎯 Interactive Setup

npx @theanikrtgiri/create-llm

You will be prompted for:

• 📝 Project name
• 🎯 Template (NANO, TINY, SMALL, BASE)
• 🔤 Tokenizer type (BPE, WordPiece, Unigram)
• 🔌 Optional plugins (WandB, HuggingFace, SynthexAI)

⚡ Quick Mode

# Specify everything upfront
npx create-llm my-llm --template tiny --tokenizer bpe --skip-install

Project Structure

my-llm/
├── 📁 data/
│   ├── raw/              # Your training data goes here
│   ├── processed/        # Tokenized data (auto-generated)
│   ├── dataset.py        # PyTorch dataset classes
│   └── prepare.py        # Data preprocessing script
│
├── 📁 models/
│   ├── architectures/    # Model implementations
│   ├── __init__.py
│   └── config.py        # Configuration loader
│
├── 📁 tokenizer/
│   ├── train.py         # Tokenizer training script
│   └── tokenizer.json   # Trained tokenizer (auto-generated)
│
├── 📁 training/
│   ├── train.py         # Main training script
│   ├── trainer.py       # Trainer class
│   └── dashboard/       # Live training dashboard
│
├── 📁 evaluation/
│   ├── evaluate.py      # Model evaluation
│   └── generate.py      # Text generation
│
├── 📁 plugins/          # Optional integrations
│
├── 📁 checkpoints/      # Saved models (auto-generated)
│
├── 📄 llm.config.js    # Main configuration file
├── 📄 requirements.txt # Python dependencies
├── 📄 chat.py         # Interactive chat interface
├── 📄 deploy.py       # Deployment script
└── 📄 README.md       # Project documentation

Complete Workflow

1️⃣ Create Your Project

npx @theanikrtgiri/create-llm my-llm --template tiny --tokenizer bpe
cd my-llm

2️⃣ Install Dependencies

pip install -r requirements.txt

3️⃣ Add Your Data
Place your text files in data/raw/:

# Example: Download Shakespeare
curl https://www.gutenberg.org/files/100/100-0.txt > data/raw/shakespeare.txt

# Or add your own files
cp /path/to/your/data.txt data/raw/

4️⃣ Train Tokenizer

python tokenizer/train.py --data data/raw/

5️⃣ Prepare Dataset

python data/prepare.py

6️⃣ Start Training

# Basic training
python training/train.py

# With live dashboard (recommended!)
python training/train.py --dashboard
# Then open http://localhost:5000

# Resume from checkpoint
python training/train.py --resume checkpoints/checkpoint-1000.pt

7️⃣ Evaluate Your Model

python evaluation/evaluate.py --checkpoint checkpoints/checkpoint-best.pt

8️⃣ Generate Text

python evaluation/generate.py \
  --checkpoint checkpoints/checkpoint-best.pt \
  --prompt "Once upon a time" \
  --temperature 0.8

9️⃣ Interactive Chat

python chat.py --checkpoint checkpoints/checkpoint-best.pt

🔟 Deploy

# To Hugging Face
python deploy.py --to huggingface --repo-id username/my-model

# To Replicate
python deploy.py --to replicate --model-name my-model

Advanced Features

Live Training Dashboard

Monitor training in real-time with a web interface, providing insights into loss curves, learning rate schedules, and resource usage.
Start the dashboard using:

python training/train.py --dashboard

Model Comparison

Quickly compare multiple trained models for performance and metrics:

python compare.py checkpoints/model-v1/ checkpoints/model-v2/

Checkpoint Management

Utilizes automatic checkpoint management to save the best model based on validation loss, maintain configurable checkpoints, and resume training from any checkpoint.

Custom Plugin Development

Create custom plugins to extend functionality, enhancing the training experience or integrating with other services.

Best Practices

Data Preparation

• Data Quantity:
  • NANO: 100+ examples
  • TINY: 1,000+ examples
  • SMALL: 10,000+ examples
  • BASE: 100,000+ examples
• Data Quality:
  • Use clean, well-formatted text with utmost consistency in encoding (UTF-8).

Training Tips

• Avoid Overfitting: Monitor perplexity during training and adjust model complexity as needed.
• Optimize Training: Use smaller models for smaller datasets to attain reliable results.

Troubleshooting

Common issues may arise, such as vocabulary size mismatches or memory errors. Refer to detailed guidance in the documentation for resolution strategies.

Getting Help

• 📖 Full Documentation
• 💬 Discord Community
• 🐛 Report Issues
• 📧 Email Support

Roadmap

Future releases aim to include additional model architectures, distributed training support, model quantization tools, and a web UI for easier project management, along with more deployment targets.

Acknowledgments

Utilizes contributions from numerous open-source tools and the LLM community, enhancing usability and functionality.