D-Adaptation

Overview

The dadaptation repository introduces a suite of advanced adaptive optimization algorithms specifically designed for neural network training. It includes implementations of the DAdaptAdam, DAdaptSGD, DAdaptLion, DAdaptAdan, and DAdaptAdaGrad optimizers, each capable of dynamically adjusting learning rates based on observed gradient statistics. This dynamic behavior enhances training performance and stability across various scenarios.

Optimizers

DAdaptAdam

The DAdaptAdam optimizer enhances the traditional Adam algorithm by introducing a flexible scaling mechanism that adapts during training. Key features include a separate scaling accumulator and a dynamic scaling factor (d₀) that calibrates the effective learning rate based on gradient variability. This promotes more efficient gradient updates and robustness through options like bias correction and decoupled weight decay.

Parameters Include:

• learning_rate: Base learning rate.
• beta1: Exponential decay rate for the first moment (mean).
• beta2: Exponential decay rate for the second moment (variance).
• epsilon: Small constant for numerical stability.
• weight_decay: Coefficient for weight decay regularization.
• d0: Initial scaling factor.

# Example Usage
optimizer = DAdaptAdam(
    learning_rate=1.0,
    beta1=0.9,
    beta2=0.999,
    epsilon=1e-8,
    weight_decay=0.0,
    d0=1e-6
)

DAdaptSGD

The DAdaptSGD optimizer presents an adaptive version of stochastic gradient descent that modifies its effective learning rate based on accumulated gradient statistics. This optimizer operates effectively with momentum and decoupled weight decay, contributing to its reliability in handling deep learning models.

Key Features Include:

• learning_rate: Base learning rate.
• momentum: Momentum factor for smoothing updates.
• d0: Initial scaling factor adapted to gradient data.

# Example Usage
optimizer = DAdaptSGD(
    learning_rate=1.0,
    momentum=0.9,
    d0=1e-6
)

DAdaptLion

The DAdaptLion optimizer is a variant that combines a sign-based update rule with an exponential moving average (EMA) for dynamic scaling based on gradient statistics. This optimizer allows for adaptive learning rates while optionally incorporating decoupled weight decay for performance enhancement.

Configuration Options Include:

• learning_rate: Base learning rate.
• weight_decay: Weight decay coefficient.
• d0: Initial adaptive scaling factor.

# Example Usage
optimizer = DAdaptLion(
    learning_rate=1.0,
    weight_decay=1e-2,
    d0=1e-6
)

DAdaptAdan

The DAdaptAdan optimizer extends the Adam algorithm by maintaining separate averages for gradients and their squared values. This is designed to respond effectively to changes in gradient dynamics while offering the flexibility for decoupled weight decay.

Parameters Include:

• learning_rate: Base learning rate.
• beta1: Decay rate for first moment estimates.
• beta2: Decay rate for gradient differences.

# Example Usage
optimizer = DAdaptAdan(
    learning_rate=1.0,
    beta1=0.98,
    beta2=0.92,
    d0=1e-6
)

DAdaptAdaGrad

The DAdaptAdaGrad optimizer enhances the classical AdaGrad method by dynamically adjusting its update scale based on the squared gradient accumulators and their differences. This approach is particularly suited for sparse gradient optimizations and supports momentum and decoupled weight decay features.

Core Features Include:

• learning_rate: Base step size for updates.
• momentum: Momentum factor to smooth updates.
• d0: Initial adaptive scaling factor.

# Example Usage
optimizer = DAdaptAdaGrad(
    learning_rate=1.0,
    momentum=0.9,
    d0=1e-6
)

Conclusion

The dadaptation optimizer collection provides versatile and powerful tools for optimizing deep learning models, catering to diverse training needs and scenarios. Each optimizer is designed to enhance performance through dynamic adjustment mechanisms and accommodates a range of parameter configurations.