Von Neumann vs neural: two ways to compute
For 64 years, computing meant a CPU fetching instructions from memory and executing them one at a time. Neural networks are a different machine. The shift matters because the software model is what changes, not just the hardware.
The von Neumann bottleneck is the round trip between processor and memory. The neural-network workload converts that bottleneck into a fundamentally different system: weights as state, matrices as instructions, tokens as output.
The von Neumann machine
In 1945, mathematician John von Neumann described the logical layout of a general-purpose electronic computer in his paper, "First Draft of a Report on the EDVAC". The design split the computer into distinct organs: the Central Arithmetic part (what we now call the Arithmetic Logic Unit, or ALU), the Central Control part (the Control Unit), the memory unit, and the input/output channels. Instead of building physical wiring pathways for each new calculation, instructions and data would share the same physical memory space, read over a shared bus. This stored-program concept was the breakthrough.
Every general-purpose processor since, from the IBM System/360 in 1964 to the CPU in your current laptop, is a von Neumann machine. The CPU operates on a simple, serial fetch-decode-execute cycle: the control unit reads an instruction from memory using the Program Counter, decodes it into control signals, schedules execution on registers and the ALU, writes the results back, and moves to the next instruction. The serial execution loop is the basic unit of work.
This separation of processor and memory was a dramatic step forward from earlier machines like the ENIAC, which required technicians to manually rewire plugboards and cables to reprogram it. With the stored-program model, programs became software, stored in the same memory as data, allowing the machine to switch between calculating artillery tables and running business ledger operations instantly.
Instructions and data travel the same path. The CPU cannot fetch the next instruction while writing a result. This is the von Neumann bottleneck.
Code lives in the same memory as data. The machine can modify its own instructions, which makes general-purpose computing possible.
Fetch, decode, execute, store, repeat. Every modern trick (pipelining, caches, out-of-order execution) exists to hide the cost of this cycle.
The bottleneck: a quantified latency wall
The design has a critical flaw: the processor and memory are physically separate, meaning every instruction and every piece of data must travel across a single shared system bus. Over the last four decades, the processing speed of logic gates has grown exponentially, but the speed of retrieving data from main memory (DRAM) has improved at a crawl. The result is the "memory wall" or the von Neumann bottleneck: a fast CPU constantly sitting idle, starved of instructions and data.
To understand the scale of this bottleneck, we must quantify the latency gap. CPU registers, which live directly inside the execution units, respond in under 1 nanosecond (sub-nanosecond, or a single clock cycle at 4 GHz). A lookup to the L1 cache takes about 1 nanosecond. The L2 cache takes 3 to 10 nanoseconds, and the L3 cache takes 10 to 40 nanoseconds. Accessing main memory (DRAM) takes 60 to 100 nanoseconds. At 4 GHz, a CPU cycle is 0.25 nanoseconds. That means a single memory fetch requires the CPU to wait for 240 to 400 clock cycles. DRAM is not slightly slower than registers: it is 100x to 200x slower.
From 1985 to 2005, CPU clock speeds grew at roughly 50% per year, while DRAM latency improved at only about 7% per year. To mask this widening gap, CPU architects developed increasingly complex latency-hiding mechanisms: multi-level cache hierarchies, branch prediction (predicting which direction a branch instruction will take), out-of-order execution (executing instruction steps out of sequence when their data is ready), and speculative execution. These workarounds consume up to 80% of a modern CPU core's transistors and thermal budget, all to patch a single structural bottleneck.
How the bottleneck was solved: Harvard architecture and caches
The earliest alternative to this design was the Harvard architecture, pioneered on the Harvard Mark I electro-mechanical computer in 1944. Instead of sharing a single memory space and bus, a Harvard machine uses physically separate memories and buses for instructions and data. This allows the processor to fetch an instruction and read or write data simultaneously, avoiding bus contention entirely.
However, pure Harvard architecture is expensive to build and rigid to program, because unused instruction space cannot be easily reallocated for data storage. To solve this, modern computer engineering settled on a hybrid design called the Modified Harvard Architecture. To a programmer, the system appears as a unified von Neumann machine with one shared memory address space. But inside the CPU package, the L1 cache is physically split into an L1 Instruction Cache (L1I) and an L1 Data Cache (L1D). Each cache has its own independent connection to the control unit and execution stages, providing the parallel throughput of Harvard architecture at the core while maintaining the programming convenience of von Neumann architecture at the system level.
Alongside split caches, processors use hardware pipelining to overlap different stages of execution (fetching a new instruction while decoding the previous one and executing the one before that), and parallel processing across multiple cores to divide workloads. While these features optimize general-purpose CPU compute, they fail when a workload consists of massive, simple mathematical operations that read data sequentially rather than branching unpredictably.
The neural network machine: a different math
A deep learning workload is mathematically simple but structurally massive. A forward pass in a neural network consists almost entirely of multiplying a matrix of inputs by a matrix of learned weights, adding a bias vector, and applying a non-linear activation function (such as ReLU or GELU). The "program" is not a complex sequence of conditional jumps or branches; the program is the static weight matrix itself. The "instruction" is a General Matrix Multiply (GEMM) operation.
This workload flips the hardware design requirements. A general-purpose CPU devotes massive transistor area to latency-hiding machinery (deep caches, branch predictors) because it must execute unpredictable branching code fast. A neural network has perfectly predictable, linear data flow: the execution units know exactly what math they will perform hundreds of steps in advance. Consequently, a deep learning accelerator drops the complex branching circuitry and fills that silicon space with thousands of simple arithmetic logic units (ALUs) running in lockstep. This is a GPU or a TPU: a wide tensor factory rather than a fast serial executor. An Nvidia Blackwell GPU has over 20,000 arithmetic cores, compared to a server CPU's 64 or 128 cores.
This structural shift also changes the memory access pattern. Instead of random-access pointer-chasing, a neural network streams gigabytes of weight parameters in bulk. The bottleneck shifts from memory latency (how many nanoseconds to fetch a single byte) to memory bandwidth (how many terabytes of weight parameters can be fed to the execution units per second). This is why High Bandwidth Memory (HBM) is critical: by stacking DRAM dies vertically and connecting them to the processor with a wide silicon interposer, HBM3e achieves up to 3 TB/s of bandwidth, compared to ~50 GB/s for standard DDR5 server memory.
Why this is "computing reinvented"
In a von Neumann world, software is code written by human developers, compiled into static binary instructions, and shipped to a user. The user interacts with a frozen, deterministic artifact. In a neural-network world, the model generates responses in real time, contextually aware and responsive to intent. The same model can write code, analyze a financial model, or compose a strategy memo depending on the prompt.
Jensen Huang, in his Stanford CS153 lecture, described this as the first true reinvention of computing in 64 years. The shift is not just about replacing one chip socket with another. It fundamentally alters what software is, how it is developed through data training rather than coding, how it is served in real time, and what business models are built on top of the computing stack.
Source: Jensen Huang, Stanford CS153 Frontier Systems lecture, April 30, 2026 (https://cs153.stanford.edu/)