Reasoning & Test-Time Search (Inference Compute)
Scaling compute at inference time. Moving from instant feed-forward responses (System 1) to search, verification, and self-correction (System 2).
Test-time search increases latency and inference costs. The challenge is balancing the compute cost per query against the value of the correct reasoning path.
System 1 vs. System 2 thinking
Traditional LLM generation is "System 1": it runs a fixed forward pass, writing the next word instantly using quick, intuitive heuristics. If the model makes an error in token 5, it cannot go back and rewrite it without reproducing the whole text.
In contrast, "System 2" reasoning models scale compute during inference. The model can generate multiple lines of thought (thoughts), stop to verify its steps, backtrack if it hits a contradiction, and explore alternative solution paths before printing the final answer.
Process-Supervised Reward Models (PRMs)
Aligning reasoning requires a change in evaluation. Outcome-supervised reward models (ORMs) only grade the final answer (e.g. Correct/Incorrect), which rewards correct answers reached via faulty logic.
Process-supervised reward models (PRMs) score every individual step of the reasoning trace. This forces the model to maintain logical consistency throughout its chain of thought, significantly reducing hallucinations in multi-step math, code, and planning.
Inference scaling: The new scaling laws
As pre-training data hits the data wall, the frontier scaling vector is shifting to test-time compute. By letting models search, generate longer chains of thought, and verify paths at inference time, we can trade compute directly for accuracy.
A query that previously took 100 milliseconds of compute can scale to run for 10 seconds, exploring thousands of reasoning trees to solve highly complex tasks.