fer

The Fractured Entangled Representation Hypothesis (FER) is a pivotal exploration into the influence of network architecture on internal representations in AI systems. This repository includes the code and supporting data for a position paper that critically addresses whether scaling up neural networks truly enhances internal representations or simply leads to better performance, which may not be indicative of improved representational quality.

Overview

Modern AI advancements often rely on the scaling of neural networks. However, the assumption that enhanced performance correlates with superior internal representation is challenged in this research. The work compares neural networks evolved through an open-ended search with those trained using traditional stochastic gradient descent (SGD) on a simple image generation task. This unique setting allows for easy visualization of each hidden neuron's contributions to output behavior, revealing significant differences in representations.

Key Findings

The study's findings reveal a fascinating distinction between the two network types:

• SGD-trained networks tend to exhibit what is termed "fractured entangled representation (FER)", indicating a disorganized internal state.
• Conversely, evolved networks display a more coherent structure, achieving what is described as a "unified factored representation (UFR)".

This contrast suggests that FER may undermine essential capabilities in large AI models, such as generalization, creativity, and ongoing learning. Understanding and mitigating FER is therefore proposed as a critical area for future research in representation learning.

Visualizations

The repository contains various visualizations, including intermediate feature maps and weight sweeps, which provide insight into the internal workings of both network types. Significant images include:

Code Functionality

The code contained within this repository allows for:

• Loading Picbreeder genomes
• Layerizing these into a multi-layer perceptron (MLP) format
• Training a SGD network to replicate generated outputs
• Visualizing internal representations
• Conducting weight sweeps and visualizing outcomes

Getting Started

A Google Colab notebook is provided for easy access and quick experimentation. Users can initiate the environment by running:

git clone https://github.com/akarshkumar0101/fer

This repository serves as a robust resource for researchers interested in deep learning representations and their implications. Additional assets and details can be accessed in the repository's structure and supplementary files.