AI’s Scaling Wall: Not the End, but a Beginning

Some say it’s an illusion, while others insist it’s real. Without doubt, in the year following ChatGPT’s emergence in 2020, AI development progressed at an astonishing pace. However, by 2024, we can clearly sense a slowdown in AI advancement.

On one hand, new models are being released less frequently; on the other hand, these new models aren’t showing significant improvements in performance. This explains why discussions about OpenAI hitting a wall have become increasingly common since the second half of this year.

We see various perspectives such as:
“Although OpenAI hasn’t hit the wall yet, given AI’s development pace, hitting the wall is inevitable.”
“AI’s slowdown indicates that scaling laws have become ineffective, making further expansion impossible.”
Or as Sam himself stated: “The wall doesn’t exist.”

Nevertheless, for frequent AI users, we can visibly observe that new models aren’t showing significant improvements, while the overall cost of using AI is increasing.

From my perspective, while the principles of scaling laws haven’t become ineffective, the wall is indeed real. This deceleration at OpenAI might create opportunities for other AI companies, potentially allowing them to surpass OpenAI at some point in the future. Below are my thoughts and predictions.

Scaling law

The principle of scaling laws was first introduced in a paper titled “Scaling Laws for Neural Language Models” by Kaplan et al.

Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute¹ used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

This is easily understood from the graph: a large language model’s performance improves with increases in model size, dataset size, and computational resources used for training.

This explains why many AI companies describe AGI as the future – theoretically, with sufficient datasets and computational resources, it’s possible to train artificial general intelligence(AGI) that surpasses human capabilities.

This also explains why, since OpenAI launched ChatGPT, other companies have started raising funds to launch their own large models, as we’ve witnessed the power of scaling laws through the performance improvements from GPT-2 to GPT-3.

For capital markets and entrepreneurs, the path seems straightforward. OpenAI has already proven this approach works. The dataset barrier isn’t really an issue – most large models are trained on public internet data. The computing power challenge can be solved simply by purchasing GPUs.

In essence, OpenAI has provided a working blueprint for many companies. As the Chinese saying goes, “crossing the river by feeling the stones” – other companies can achieve good results by following this proven path.

Hitting the wall doesn’t mean scaling laws have failed.

Top AI models have shown a clear slowdown in the last 6 months compared to their rapid progress over the previous two years. This trend is widely reported in tech media, including Business Insider.

Importantly, this slowdown doesn’t invalidate scaling laws – it reinforces them. The bottleneck clearly lies in dataset limitations and computational power constraints.

Datasets

Ilya also mentioned a similar perspective at NeurIPS 2024

He views data as the oil of AI and believes that existing data isn’t growing exponentially.

Computational Power

What about our computational power? Has it reached a bottleneck?

Musk’s recent mention of a supercomputing center on X suggests that major AI companies haven’t yet reached peak computational capabilities.

The recent All-In Podcast confirmed this: “While building X AI, Elon Musk devised a radically different approach to building data centers, achieving coherence across over 1,000,000 GPUs. This is a world-first implementation – something Google and Meta’s engineers said couldn’t be done.”

This suggests several possibilities:

1. The current slowdown in large model growth provides opportunities for new open-source AI companies to catch up with OpenAI. The moat around large AI companies might not be as deep as we imagine (or might not exist at all).
2. Next-generation AI companies, led by X AI and deepseek, could surpass current leading models through engineering and architectural optimizations.
3. As mentioned in the paper “The Platonic Representation Hypothesis,” all large models might converge toward a single world model, with only monopolistic leaders surviving and creating super AGI (a concerning future that I’ll detail in my next article).

Newton’s reflector

The real bottleneck may not be the laws, but engineering.

This reminds me of telescopes in Galileo’s era. People discovered that building longer telescopes allowed them to see further, eventually constructing telescopes over 40 meters long.

Then Newton, using basic principles of optical design, successfully shortened the telescope tube many times over while greatly improving its performance.

Newton didn’t change the telescope’s scaling laws – instead, he leveraged the principles of light reflection to dramatically improve performance while reducing size.

Breaking through “the wall” often isn’t about simply scaling up, but about architectural innovation. For AI, the next breakthrough might similarly require revolutionary innovations in fundamental architecture, rather than just throwing more computing power at the problem.

Future breakthroughs may lie in:

• Novel neural network architectures
• More efficient training methods
• New computing paradigms

Just as Newton’s telescope ushered in a new era of astronomy, the next revolutionary innovation in AI might already be taking shape.