It’s not just about scale
Intelligence is about more than just compute and data.
There’s a lot of handwaving that goes on in the world of large language models and the path towards the mystical “AGI”. It usually goes like this:
Progress has been rapid…
There’s no reason to believe progress won’t continue to improve rapidly as we add more data and more compute…
Voila! This will eventually get us AGI
Yes, these large language models have achieved impressive results in text generation, data processing, image creation and more — there’s no doubt about that. But it’s crucial to remember that these achievements, while remarkable, may not be the whole story when it comes to building truly intelligent systems.
There are few, if any, reasons to believe that “scale” (moar data, moar compute, etc.) is the core constraint to building digital intelligence. This is especially evident if consider how intelligence emerged in nature.
Intelligence is navigating uncertainty
Fundamentally, intelligence is about navigating an uncertain world. Navigating an uncertain world is about developing and testing causal hypotheses about how the world works. And developing causal hypotheses about how the world works is about much more than text generation — it requires an ability, as a dynamic entity in a dynamic world, to continuously adapt and learn in pursuit of accomplishing one’s goals.
Consider how children learn to ride bikes: They don’t simply absorb information from a manual. They form predictions about how to ride a bike — how to balance and pedal — then they take actions to test these predictions, and refine their internal models based on the results. This iterative cycle of prediction, perception, action, and learning is the hallmark of natural intelligence.
Active Inference and Natural Intelligence
Karl Friston formalizes it in his theory of Active Inference: intelligence emerges from an ongoing cycle of prediction, perception, action, and learning. It’s a continuous process that’s about much more than text or image generation; it’s about how we perform in specific scenarios where data may be limited; it’s not about finding the global average across some mass amount of information.
Crucially, this process involves understanding causality — not just observing correlations, but grasping how actions lead to specific outcomes. Current AI systems excel at finding patterns in data, but they struggle with the kind of causal reasoning that comes naturally to humans. They can’t design and execute experiments to test their understanding of the world, a key aspect of both scientific inquiry and everyday human learning.
To be clear — this is not to refute the possibility that we may one day build digital forms of intelligence that qualify as “AGI”. (However, it’s worth noting there’s no clear consensus on what exactly constitutes “AGI”… ask ten people and you’ll get thirty different answers). Rather, this should serve as a reminder that there are more unsolved problems in the landscape than “scale”. Scale may turn out to be necessary — it probably is — but it’s not all that you need.
While current AI approaches have yielded some remarkably impressive results, true intelligence likely requires more than just increasing the size of our models and datasets. We should continue exploring approaches that incorporate causal reasoning, embodied learning, and dynamic interaction with the environment.
This demands more than just amplifying our current models. Nature’s blueprint for intelligence isn’t about sheer processing power — it’s about adaptability, curiosity, and the ability to form and test ideas about an ever-changing world.
As we push forward in AI development, we must grapple with these fundamental aspects of intelligence. Our goal shouldn’t just be larger neural networks, but systems that can reason, explore, and learn with the versatility and depth (and efficiency) we see in biological intelligence.
