ClassicNeuralNetworks

Classic Neural Networks provides implementations of various traditional neural network architectures and training algorithms, serving as a comprehensive resource for understanding and utilizing foundational concepts in neural network design. The repository draws on the principles outlined in the acclaimed book by M. Hagan et al., Neural Network Design, 2nd edition, 2014.

Key Features:

• Adaptive Predictor: A simple perceptron network designed to predict outcomes based on two previous time steps.

  # Example usage of Adaptive Predictor
  python adaptive_predictor.py
  

  

• Bayesian Regularized Neural Network: Implements a Bayesian approach to neural networks, employing the Levenberg-Marquardt algorithm for training. It evaluates different activation functions including sigmoid, ReLU, ELU, and Leaky ReLU.

  

• Levenberg-Marquardt Training Algorithm: Enable training of neural networks using the Levenberg-Marquardt approach. Supported activation functions include sigmoid, ReLU, ELU, and Leaky ReLU.

  

• Multi-layer Feedforward Neural Network: A customizable multi-layer feedforward architecture for regression tasks, supporting several optimization algorithms such as gradient descent, conjugate gradient descent, and the Adam algorithm.

• Radial Basis Function (RBF): This implementation incorporates the orthogonal least squares (OLS) algorithm for the first layer and least mean squares (LMS) for the second layer of the RBF network.

• Recurrent Neural Network (RNN): Supports arbitrary recurrent connections with training performed via Backpropagation Through Time (BPTT). A Keras-like API facilitates defining layers and their connections in the RNN architecture.

  

• Experimentation: Included scripts for running various experiments, such as an RNN predicting magnetic levitation and testing the performance of different training algorithms and activation functions. The results of these experiments illustrate the comparative effectiveness of methods such as gradient descent, conjugate gradient descent, and the Adam optimization technique.

  

• Utility Functions: Comprehensive utility functions are provided for managing configurations, logging, and processing data within neural networks.

This repository thus serves as a practical guide and resource for researchers and developers interested in deep learning and neural network methodologies.