The Physics of Scale (Scaling Laws & Compute Economics)
AI capability scales predictably with compute, parameters, and tokens. Understand the power laws and optimal boundaries that dictate frontier training.
Historically, we scaled parameters faster than data. Today, the constraint is the data wall: the internet is running out of high-quality human text, forcing a shift from pre-training scale to inference-time search.
The power-law scaling curve
Performance in next-token prediction does not improve linearly; it follows a power law. OpenAI's early scaling research (Kaplan et al.) showed that cross-entropy loss drops predictably as a power function of three variables: compute budget (FLOPs $C$), model parameters ($N$), and dataset token volume ($D$).
As long as you scale these factors in harmony, performance continues to climb, and scaling remains the most reliable engine for unlocking new emergent reasoning capabilities.
The Chinchilla correction (Compute-optimal training)
DeepMind's Chinchilla paper (Hoffmann et al.) corrected the optimal scaling ratio. They showed that previous models (like GPT-3) were under-trained on data and over-allocated on parameters. For a compute-optimal training run, parameters and tokens should scale in equal proportion: $N propto D$.
To train a model optimally, you need about 20 tokens of training data for every 1 parameter in the model. A 70B model requires at least 1.4 trillion tokens to be compute-optimal.
Overtraining for inference economics
While Hoffmann defined compute-optimal training, real-world deployments prioritize inference economics. If a model will be run billions of times, it is cheaper to overtrain a smaller parameter model (e.g. Llama-3 8B trained on 15 trillion tokens, a ratio of ~1800:1) to squeeze maximum capability into a cheaper footprint.
This shifts the bottleneck: we overpay on pre-training compute to save orders of magnitude on serving costs.
The data wall and synthetic bootstrapping
The biggest bottleneck to scaling laws today is the data wall. The world is running out of clean, human-generated text. To keep scaling, frontier labs are transitioning from passive internet scrapings to high-quality synthetic data, verified code execution loops, and bootstrap pipelines that extract reasoning traces.