In Book VII of “The Republic,” Plato describes this allegory: a group of prisoners have been chained in an underground cave since birth, their necks and legs bound, able only to face the cave wall. Behind them stands a low wall, and beyond it lies a pathway with torches where people walk carrying various objects. These prisoners, throughout their lives, can only see the shadows cast on the wall and hear echoes, taking these for the whole of reality.
If one of the prisoners were to be freed, he would initially be pained and confused by the bright light, but after adapting, he would be able to see the real world. When he returns to the cave to tell the other prisoners about the truth outside, he would be met with mockery and rejection.
This represents the core tenets of Plato’s Theory of Forms: There exist two worlds – one composed of Forms, and another of matter.
The world we inhabit is merely a projection of the world of Forms.
“No one has ever seen a perfect circle or a perfect straight line, yet everyone knows what a circle and a line are.”
Forms are the paradigms of phenomena
Phenomena are mere copies of the Forms
In his description, philosophers are the first to willingly leave the cave, and their sacred mission is to guide humanity from darkness into light.
This thought laid the foundation for Western philosophy for millennia to come, while also sparking a series of philosophical debates – phenomenon versus essence, universal versus particular, reason versus sensibility, and so on.
This ancient philosophical question about “reality” versus “appearance” has found new inspiration in contemporary artificial intelligence research. How do modern neural networks understand and represent the world? Are they, like the prisoners in the cave, only able to access “projections” of reality?
This leads to an important research hypothesis:
The Platonic Representation Hypothesis
On August 11, 2024, Phillip Isola, then an Assistant Professor at MIT, gave a public lecture at BMM explaining this paper.
Here he proposed a central thesis – Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.
Evidence of Convergence
How should we understand this convergence?
This bold conjecture isn’t without basis—Phillip Isola and his team, along with other researchers, have provided compelling evidence supporting the “Platonic Representation Hypothesis” through a series of experiments and analyses. These findings suggest a fascinating possibility: although neural networks encounter different “shadows,” they ultimately learn to capture the underlying structure of the same “real world.”
Gabor-like filters
The emergence of Gabor-like filters in the early layers of convolutional neural networks, similar to those found in biological visual systems, provides compelling evidence that there exists an optimal, universal way to extract basic image features such as edges and orientations.
This figure illustrates a crucial connection between biological visual systems and artificial neural networks.
1. On the left is the classic 1959 experiment by Hubel and Wiesel. They placed electrodes in cats’ visual cortex to record neuronal activity and discovered that neurons in the visual cortex are most sensitive to lines of specific orientations. These neurons effectively function as orientation-selective filters.
2. The middle section depicts Gabor filters, which are mathematical tools designed to detect edges and textures of specific orientations, functioning similarly to neurons in the visual cortex.
For example, when observing a line tilted at 45 degrees, the filter corresponding to that orientation produces the strongest response.
3. On the right are the filters learned by the first layer of AlexNet (a classic CNN). Interestingly, these filters automatically learn patterns resembling Gabor filters. They too detect edges and textures of different orientations.
This demonstrates that artificial neural networks spontaneously develop feature extraction mechanisms akin to those of biological visual systems. This convergence phenomenon not only suggests that this might be the optimal approach for processing visual information but also provides compelling evidence that different systems tend toward similar representations.
Rosetta Neurons:
The concept of “Rosetta Neurons” is particularly striking. These neurons act like translators between different languages, being activated by the same patterns across different visual models. It’s similar to how people describe the same thing in different languages:
In Chinese: “猫的眼睛”
In English: “cat’s eyes”
In French: “les yeux du chat”
While the expressions differ, they point to the same concept. Although these models “speak different languages” (having different architectures and training methods), their ways of understanding the world are remarkably similar.
This suggests that even with different architectures and training approaches, these models develop aligned, mutually comprehensible internal representations.
Model Stitching:
When the intermediate layer representations from one model can be directly inserted into another model while maintaining good performance, it directly demonstrates that the representations learned by different models are compatible and interoperable.
Alignment Increases with Scale and Performance:
More powerful, better-performing models tend to show higher degrees of representational alignment. This suggests that as models’ capabilities increase, they increasingly tend to learn representations closer to “reality” rather than being limited to specific training objectives or datasets.
Convergence Across Modalities:
The representational alignment between vision and language models is particularly exciting. As both visual and language models become better at their respective tasks, their internal representations grow increasingly similar. This suggests that whether understanding the world through vision or language, systems ultimately touch upon common concepts and structures.
Hypotheses Derived from Evidence
Multitask Scaling Hypothesis
As the number of tasks a model needs to handle simultaneously increases, the number of representations capable of handling all tasks actually decreases.
This concept can be intuitively understood through a Venn diagram: the intersection of solution spaces for different tasks (such as Task 1 and Task 2) represents the representation space that can solve multiple tasks simultaneously, and this intersection is necessarily smaller than the solution space for any single task. This hypothesis is supported by both the “Anna Karenina Principle” and the “Contravariance Principle,” which respectively demonstrate that successful deep networks tend to learn similar internal representations, and that stronger constraints lead to fewer possible solutions. This suggests that when we train models to solve more tasks, they may be forced to converge toward more fundamental representations, which might explain why neural networks with different architectures develop similar “Rosetta Neurons.”
The Capacity Hypothesis
Larger models are more likely to converge to shared representations. This concept can be understood through the evolution of hypothesis spaces: as we scale up model architectures, the hypothesis spaces of different models (Hypothesis space 1 and 2 shown in the figure) expand, leading to larger overlapping regions.
The concentric circles in the figure represent loss function contours, and the star-marked optimal solutions that may be far apart in smaller models naturally converge toward a common optimal region when model capacity is increased. This hypothesis suggests that as models grow in scale, different architectures may spontaneously discover representations closer to “reality,” which aligns with phenomena we observe in large language models and vision models.
The Simplicity Bias Hypothesis
The Simplicity Bias Hypothesis reveals a fundamental characteristic of deep networks: they naturally tend to seek simple fitting solutions to data, and this preference strengthens as model scale increases. The left side of the figure shows two regions in the hypothesis space: one is the set of all functions that solve the tasks (purple region), and the other is the set of simple functions (blue region).
The simplicity bias (shown by arrows) drives models to prioritize the intersection of these two sets. The experimental results on the right demonstrate representation patterns across networks with different depths and activation functions (linear, tanh, relu, leaky relu, selu, gelu, siren), showing remarkable similarities. This supports the notion that as models grow larger, their solution space converges toward simpler, more fundamental representations.
This hypothesis provides a theoretical foundation for understanding why large models can capture the basic laws of the real world.
Possibilities
These hypotheses all point to an intriguing possibility:
Although neural networks’ training data are like shadows on the cave wall, the representations they ultimately learn somehow transcend these limitations, reaching toward deeper, more fundamental models of the world.
This is the core idea of the “Platonic Representation Hypothesis” – neural networks, like philosophers who have left the cave, are striving to understand the world of “Forms,” even though they initially can only see projections of “phenomena.”
Modern Mapping of AI Systems to the Cave Metaphor
Several parallel relationships can be observed here:
Training data corresponds to the shadows on the cave wall, while model limitations correspond to the prisoners’ chains.
Training data is like these shadows—they are projections of the real world after digitization, sampling, and annotation. Whether it’s selectively collected photographs in image datasets or human biases in language model training texts, they are like the dancing shadows on the wall, both incomplete and subjective. Even real-time data streams remain constrained by sensor capabilities and data collection methods, just as the cave’s firelight can only project partial features of objects.
Meanwhile, model architectures play the role of “chains,” determining how AI systems “perceive” and “understand” this data. Convolutional neural networks inherently carry a bias toward processing local features, while the attention mechanism in Transformers presupposes specific ways of processing information. Different architectural choices are like different shackles—while they all enable models to “see” something, they simultaneously limit their field of vision.
Philosophical Significance of Representational Convergence
For this surprising phenomenon of representational convergence, we face three profound philosophical interpretations:
First, there might indeed exist some objective structure of a “world of Forms,” just as Plato envisioned, where all intelligent systems—whether biologically evolved brains or artificially trained neural networks—are approaching this ultimate reality through different paths.
The second, more conservative interpretation is that this convergence merely reflects the statistical laws of the physical world, similar to how different climbers will eventually find similar paths to reach a mountain peak. The consistency in these representations might simply stem from the coincidental similarity of optimal problem-solving strategies.
The third possibility, with greater philosophical depth, is that the representational convergence we observe might itself be another layer of “shadows”—the tools and methods we use to detect and understand this convergence carry their own limitations, just as prisoners, even when freed from the cave, might still only perceive projections of a higher-level reality.
This reminds me of Stephenson’s novel “Anathem,” where events occurring in the protagonist’s world create projections across multiple universe civilizations, leading to visits from “cousin aliens.” In Stephenson’s writing, the causal relationships in Anathem unfold in the form of a DAG (Directed Acyclic Graph).
Could there be a possibility that
The representational convergence of different models isn’t simply “convergence,” but rather follows a directed acyclic causal structure:
- Lower-level representations determine the possibilities for higher-level representations
- Once a representational pathway is formed, it influences the direction of subsequent development
- Models with different starting points may take different paths, but ultimately intersect at certain critical nodes in the DAG
Perhaps there exists some universal “cognitive DAG” that determines the developmental pathway that all intelligent systems must follow.
The Philosopher King and AGI
In “The Republic,” Plato introduces the concept of the “Philosopher King” – only true philosophers can be rulers of the ideal state.
Until philosophers are kings, or the kings and princes of this world have the spirit and power of philosophy … cities will never have rest from their evils,—no, nor the human race, as I believe,—and then only will this our State have a possibility of life and behold the light of day
In my view, this forms an interesting historical parallel. The main criticisms of the Philosopher King concept coincidentally apply to current expectations of Super AGI:
Aristotle points out in “Politics”: perfect rulers cannot exist because human nature itself is imperfect.
“For the law cannot foresee all cases, and the imperfection of human nature means that no one can rule with complete justice.”
This parallels Nick Bostrom’s warning in “Superintelligence” – expecting a perfect AGI system is itself a dangerous form of idealism.
Popper’s harsh criticism in “The Open Society and Its Enemies” asks: “Who should rule? How can we ensure they won’t abuse their power?” This mirrors Stuart Russell’s AI control problem: “How can we ensure that systems with superintelligence will act in accordance with human values?”
Just as “The Prince” points out that the ideal ruler is difficult to achieve because real politics is full of compromises, this echoes Yudkowsky’s concerns about AGI alignment: “We can’t even precisely define human values, how can we expect AI to execute them perfectly?“
This parallelism reminds us that, just as the Philosopher King was never realized, a perfect AGI might also be a utopian fantasy.
Ancient people hoped to delegate governance responsibilities to “perfect rulers,” while today’s people expect AGI to “perfectly solve” all problems.
Isn’t this just another form of fantasy and escapism? More importantly, this messianic complex about super AGI might lead us to an even more terrifying future, which I will discuss in the next chapter.
Reflections
When this paper appeared, it sparked considerable discussion, and as its author mentioned in the later part of his talk, this hypothesis has certain limitations. For example, in specific domains like autonomous driving or protein folding, specialized large models might perform better.
Of course, I have no intention to discuss this hypothesis in these domains. I believe that even at this current juncture, the philosophical and scientific implications of this paper are underestimated.
While walking one day, I suddenly realized that Plato’s choice of dialogue form to convey his ideas was no coincidence. In “The Republic,” he carefully reconstructed Socrates’ dialogues, using this interactive format to present complex philosophical ideas. This methodology bears striking similarities to modern large language models:
- Plato conveyed ideas through reconstructed dialogues
- LLMs express their learned knowledge through dialogue
- Both attempt to approach some form of “truth” through interaction
Even more thought-provoking is that perhaps we need to rethink our pursuit of AGI itself.
If Plato’s Cave Allegory is a prophecy, then perhaps: it’s not humans themselves who will leave the cave, but rather these “ideational models” we’ve created that can transcend specific phenomena and reach directly to essence.