Univariate Gaussian Mixture Model Neural Network (uGMM-NN)

This repository introduces the Univariate Gaussian Mixture Model Neural Network Model (uGMM-NN). This experimental feedforward neural network architecture replaces traditional neuron operations with probabilistic univariate Gaussian mixture nodes. By parameterizing neurons with means, variances, and mixing coefficients, uGMM-NNs capture multimodality and uncertainty that standard MLP neurons cannot represent. This mixture-based view allows richer probabilistic reasoning within deep neural networks, making them especially promising as building blocks for next-generation architectures.

Model Overview

Designed as a feedforward neural network, the uGMM-NN presents a novel approach to integrating probabilistic models into neural computations. At its core, each neuron acts as a univariate Gaussian Mixture Model (uGMM), enabling a richer representation of the input data. This method allows for both complexity in modeling and robustness in learning.

Structure of the uGMM Node

A uGMM neuron, denoted as j, processes N inputs (x₁, ..., xₙ) derived from the preceding layer. It encompasses a Gaussian Mixture Model tailored to the input dimensions, featuring:

• Learnable Parameters: Each uGMM node has N components, with unique means (μⱼ,ₖ), variances (σ²ⱼ,ₖ), and mixing coefficients (πⱼ,ₖ) for every neuron.

Implementation Example

The architecture is straightforward, reflecting classic neural networks through input, hidden, and output layers. Each node in the network embodies a univariate Gaussian mixture model, amalgamating inputs to form a hierarchical structure capable of constructing complex, high-dimensional probability distributions. The following example illustrates the network architecture:

For practical applications, utilize the notebooks directory containing Jupyter notebooks that provide insights into the usage of this implementation:

• Generative Inference: Explore interactions within the Iris dataset using a uGMM trained with NLL loss in the Iris Notebook.
• Discriminative Inference: Analyze results on the MNIST dataset in the MNIST Notebook.

Future Work

Future research directions include:

• Architectural Extensions: Redesigning classical deep learning architectures, such as recurrent neural networks (RNNs) and Transformers using uGMM neurons may enable fully probabilistic, uncertainty-aware versions of these models. By replacing traditional MLP layers with uGMM layers, each neuron can propagate multi-modal hypotheses through the network, allowing richer latent representations and probabilistic reasoning at every layer. Such extensions could enhance the network’s robustness to ambiguous or noisy inputs and improve interpretability by exposing which mixture components dominate decisions.

• For a generatively trained uGMM-NN, computing the Most Probable Explanation (MPE) is challenging due to the continuous Gaussian components and complex inter-layer dependencies. While this may limit appeal for probabilistic circuit applications, for standard deep learning use cases such as replacing MLP layers with uGMM layers in discriminative networks, this limitation does not apply.

Related Research

This repository builds upon significant contributions in the field:

• Peharz et al. (2020) examine scalable learning of probabilistic circuits in "Fast and scalable learning of tractable probabilistic circuits" presented at ICML.
• Choi et al. (2020) discuss a unifying framework for probabilistic models,
• Poon and Domingos (2011) introduce sum-product networks in their IEEE Conference paper.

For reference or citation purposes, the following BibTeX entry can be used:

@software{zakeriaali2025ugmmnn,
  author = {Ali, Zakeria},
  title = {{uGMM-NN: A Deep Probabilistic Model with Univariate Gaussian Mixture Nodes}},
  url = {https://github.com/zakeria/ugmm},
  version = {0.0.