Overview of Pygrad: Lightweight Automatic Differentiation Engine in Python

Pygrad is a compact and efficient automatic differentiation engine implemented in Python, optimized for small-scale applications without excessive overhead. Its foundation is built upon the well-regarded libraries NumPy, Numba, and opt_einsum, ensuring both reliability and speed.

Key Features

• Differentiable Tensor Class: Central to Pygrad is the Tensor class, allowing seamless integration with mathematical operations. Each tensor holds its value and gradient, facilitating easy computation of derivatives through its .backward() method.
• Diverse Layer Support: Includes implementations for common neural network layers such as Dropout, Linear, and Attention, making it versatile for various AI models.
• Comprehensive Loss Functions: Supports standard loss functions like Binary Cross-Entropy (BCE) and Categorical Cross-Entropy (CCE), essential for training models effectively.
• Optimizers: Offers popular optimization algorithms including Stochastic Gradient Descent (SGD), RMSProp, and Adam.
• Model Architectures: Provides example architectures for Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), and Transformers, illustrating its capacity for handling complex models.

Usage Guide

Pygrad's tensors behave similarly to NumPy arrays, making them intuitive for those familiar with Python’s scientific computing stack. To create a tensor, simply initiate it with a desired value:

from pygrad.tensor import Tensor
x = Tensor(1)
(((x**3 + x**2 + x + 1) - 1)**2).backward()
print(x.value, x.grad)  # Outputs: 1.0, 36.0

Gradient descent can be performed efficiently using Pygrad's tensors, as demonstrated below:

for _ in range(100):
    (((x**3 + x**2 + x + 1) - 1)**2).backward()
    x.value = x.value - 0.01 * x.grad

Tensors in Pygrad accept multidimensional arrays and support operations that are compatible with NumPy's broadcasting rules:

import numpy as np
x = Tensor(np.ones((10, 20)))
y = Tensor(np.ones((20, 10)))
z1 = x @ y
z2 = x @ np.ones((20, 10))
assert np.all(z1.value == z2.value)  # True

Pygrad's library is rich enough to support the development of various machine learning models. For detailed guidance on using each component, consult the official documentation.

Contribution and Support

For those interested in contributing to the project or needing assistance, issues can be submitted via GitHub. Contributors can find additional guidance at the contribution status page. Researchers and developers making use of Pygrad are encouraged to provide a citation, acknowledging its utility in their work.