numpyGPT

numpyGPT is an implementation of a Generative Pre-trained Transformer (GPT) from scratch, utilizing only NumPy and Python’s standard library. This project emphasizes a deep understanding of the core components of neural networks, including modules, tokenizers, optimizers, and backpropagation, without relying on any automatic differentiation frameworks or tensor abstractions. The result is a clear, transparent implementation where every computation, including gradient calculations, is explicit.

Key Features

• Explicit Gradient Calculation: Understand the mechanics of backpropagation with clear visibility into every computation.
• Familiar API: The interface is designed to be reminiscent of popular deep learning frameworks, making it easier for users familiar with PyTorch or TensorFlow to adapt.
• Complete Transformer Model: The architecture includes essential transformer components such as multi-head attention, feedforward layers, and layer normalization.
• Flexible Tokenization Options: Support is provided for character-level, word-level, and Byte Pair Encoding (BPE) tokenization methods.
• Robust Testing Suite: Comprehensive tests are available to ensure the correctness of both forward and backward passes for every layer.
• Minimal Dependencies: The project solely relies on NumPy and the standard library, resulting in a lightweight setup.

Core Modules

The package comprises several functionalities, including:

• Basic building blocks: Linear, Embedding, LayerNorm, Softmax, ReLU, MultiHeadAttention, and FeedForward.
• An Adam optimizer for efficient training.
• Tokenization strategies including character-level, word-level, and BPE.
• A fully implemented GPT model based on the transformer decoder.

Code Examples

Here’s a sample implementation of a linear layer to illustrate the code style and structure:

class Linear:
    def __init__(self, in_features, out_features):
        self.W = np.random.randn(in_features, out_features) * 0.02
        self.b = np.zeros(out_features)
    
    def forward(self, X):
        self.X = X  # cache for backward
        return X @ self.W + self.b
    
    def backward(self, dY):
        self.dW = self.X.T @ dY        # gradient w.r.t weights
        self.db = np.sum(dY, axis=0)   # gradient w.r.t bias  
        dX = dY @ self.W.T             # gradient w.r.t input
        return dX

Educational Resources

The repository includes various educational documents that explain:

• Backpropagation and its implementation.
• The differences between various optimization algorithms in Optimizers.
• Tokenization methods discussed in Tokenizers.

Experimental Analysis

The project also conducts experiments to evaluate the efficacy of different tokenization strategies (Character-level, Word-level, BPE) on training transformer models using Shakespeare's text. With detailed hyperparameters and evaluation metrics, insights into model performance, output quality, and parameter efficiency are highlighted.

In summary, numpyGPT serves as an educational resource for those looking to gain a deeper understanding of neural networks and the underlying mechanics of modern language models. The hands-on implementation provides a fundamental foundation that is crucial for advancing in the field of machine learning.