Edition 3 · 01 / 08
Why Did Bigger Models Become Smarter?
Moore’s Law improved the machinery. AI also grew because we chose to use vastly more of it.
If every new AI model seems more capable, it is tempting to credit the chips. Computers get faster, therefore AI gets smarter. That explanation misses an important choice: researchers have also used many more computers, for much larger experiments.
The distinction matters. Better machinery can make an existing activity cheaper. Expanding the activity can make the total bill larger at the same time.
Two different kinds of scaling
Moore’s Law describes a historical trend in the number of components that can be economically placed on an integrated circuit. It is often shortened to a roughly two-year doubling. It is not a law guaranteeing that every application becomes twice as fast, or every datacentre half as expensive. Performance also depends on memory, communication, software and how efficiently the machinery is used. MIT’s account of computing power and innovation explains why hardware and algorithms must be considered together.
AI scaling describes something else: the resources devoted to training and running a model. OpenAI’s 2018 analysis of selected large training runs found a roughly 3.4-month doubling in compute from 2012, amounting to more than a 300,000-fold increase over its observed period. That is a historical sample, not a forecast or a description of every training run today. It nevertheless shows why semiconductor progress alone cannot explain the increase. Researchers were also willing and able to spend more computation. OpenAI, AI and compute.
What a parameter buys
A parameter is an adjustable numerical weight. Training changes those weights to improve predictions. A token is a unit of input or output, often a word fragment. Training compute is the arithmetic spent processing examples and updating the model.
Think of a richer vocabulary for describing relationships. A larger network can express more complicated patterns, but the usefulness of that vocabulary depends on what it learns and how it is taught. Parameters do not each contain an individually labelled fact.
Kaplan and colleagues found empirical power-law relationships between language modelling loss and model size, data and compute. Loss measures predictive error on a specified distribution. Lower loss is useful evidence of better prediction; it is not a direct measurement of judgement, truthfulness or business value. Those outcomes need their own evaluations. Scaling Laws for Neural Language Models.
More resources can help. Their balance matters.
Chinchilla changed the question
DeepMind’s Chinchilla study tested how to divide a fixed training budget between parameters and tokens. Its 70-billion-parameter model used the same training compute as the 280-billion-parameter Gopher, with four times as much training data, and outperformed it on a broad set of evaluations. The result challenged the assumption that the best use of a budget was simply a larger model. Hoffmann et al., Training Compute-Optimal Large Language Models.
That is an allocation result, not a permanent prescription for every model. A system used millions of times may justify more training to make each later answer cheaper. A specialist application may prioritise a different dataset. Training efficiency and lifetime operating cost are related but distinct objectives.
A gain is not the same as a guarantee
Useful behaviours can become visible when performance crosses a threshold. A model that almost always fails a task and one that sometimes completes it feel qualitatively different. Yet the headline depends on the test: exact success, partial credit and average prediction loss can tell different stories.
The practical question is therefore more demanding than “How large is it?” Ask which capability improved, on which workload, with what budget and what failure rate. A lower average error can coexist with an unacceptable mistake on the case that matters to you.
Scaling laws give engineers a way to reason about resources. They do not promise that each extra pound buys the same improvement forever. Nor do they establish an imminent wall. The next article examines why capacity itself is more complicated than a count of weights.