AI is so smart, why are its internals 'spaghetti'? We spoke with Kenneth Stanley and Akarsh Kumar (MIT) about their new paper: Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis. Co-authors: Jeff Clune Joel Lehman

"Questioning Representational Optimism in Deep Learning" reveals that AI models achieving perfect outputs may have completely broken internal representations.

The key finding: Two neural networks can produce IDENTICAL outputs while having radically different internal representations. One has clean, modular structure (UFR - Unified Factored Representation) The other is "total spaghetti" (FER - Fractured Entangled Representation)

This matters because FER potentially cripples: • Generalisation to new scenarios • Creative problem-solving • Continual learning abilities The very capabilities we need AI to excel at for real-world impact.

Evidence of FER in current models: * GPT-3 could count office supplies but failed counting animals * Image models can add extra thumbs to human hands but fail with ape hands * LLMs use different arithmetic circuits for different contexts

In well-structured networks, adjusting single weights produces meaningful changes (opening mouth, winking eye). In FER networks, the same adjustments create chaotic distortions.

Why does this happen? The *path* to the solution matters as much as the solution itself. Open-ended discovery (like Picbreeder) builds modular understanding bottom-up. Conventional SGD takes shortcuts, creating "impostor" solutions.

This is a BIG deal and the BEST way to understand why current AI has limitations (particularly around inventive creativity) * Scaling alone may not fix fundamental representation issues * Current AI might be elaborate memorisation rather than true understanding * We may need fundamentally different training approaches

Potential solutions : * Open-ended search algorithms * Better training curricula * Architectural changes promoting modularity * Methods that encourage "evolvability" of representations

The field is currently suffering from "representational optimism" - the belief that good performance implies good understanding. BUT... Are we building true intelligence or sophisticated impostors?

Check their paper here, it's WORTH your time AND... @kenneth0stanley is HIRING (this is an OPPORTUNITY OF A LIFETIME!) Research Engineer: Research Scientist:

Scan any documents, convert images into text, PDF files, etc. 👍