Adan

The Adan optimizer is a cutting-edge optimization algorithm tailored for deep learning applications, specifically enhancing training efficiency and convergence rates. Known as Adaptive Nesterov Momentum, Adan leverages advanced techniques such as adaptive gradient estimation and multi-step momentum to offer superior performance compared to traditional optimizers.

The algorithm is detailed in the paper: "Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models" (arXiv link). This implementation draws inspiration from the official repository, ensuring adherence to the core principles outlined in the original research (Adan GitHub Repository).

Key Features and Parameters:

• learning_rate (float, default=1e-3): Specifies the learning rate for optimizing the model.
• beta1 (float, default=0.98): Sets the exponential decay rate for first moment estimates.
• beta2 (float, default=0.92): Determines the decay rate for gradient difference momentum.
• beta3 (float, default=0.99): Controls the decay rate for second moment estimates.
• epsilon (float, default=1e-8): A small constant that ensures numerical stability.
• weight_decay (float, default=0.0): Defines the strength of weight decay regularization to prevent overfitting.
• no_prox (bool, default=False): Disables proximal updates during weight decay when set to True.
• foreach (bool, default=True): Enables optimization via multi-tensor operations.
• Gradient clipping options: Options available to clip gradients by their norm, individual values, or global norms via parameters clipnorm, clipvalue, and global_clipnorm.
• Exponential Moving Average (EMA): Parameters such as use_ema (bool, default=False) and ema_momentum (float, default=0.99) facilitate advanced parameter averaging techniques to stabilize training.
• Miscellaneous settings: Includes options for loss scaling in mixed precision (loss_scale_factor) and gradient accumulation steps (gradient_accumulation_steps).
• name (default="adan"): Allows customization of the optimizer's identifier.

Example Usage:

import tensorflow as tf

# Initialize the Adan optimizer
optimizer = Adan(
    learning_rate=1e-3,
    beta1=0.98,
    beta2=0.92,
    beta3=0.99,
    weight_decay=0.01,
    use_ema=True,
    ema_momentum=0.999
)

# Compile a model
model.compile(
    optimizer=optimizer,
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

# Train the model
model.fit(train_dataset, validation_data=val_dataset, epochs=10)

Leveraging the Adan optimizer can lead to improved training dynamics and faster convergence times, making it a valuable tool for deep learning practitioners seeking to enhance model performance.