Why is cooling an architecture choice?

Every watt becomes heat

Almost all electricity consumed by a data center eventually becomes heat. The compute layer turns power into answers, then the facility has to move the heat somewhere else.

That makes thermal design a first-order constraint. If heat removal fails, utilization fails. A B200 turns roughly 1.2 kW into heat on a die the size of a matchbook, and the building has to walk that heat out to the atmosphere across a chain of interfaces. Each interface resists the flow, and the whole discipline of cooling is lowering whichever resistance is worst.

The relay from junction to atmosphere

The chain has five links, and every cooling architecture is a choice about which links to keep.

• Inside the package: heat conducts from the transistors up through the silicon and a thermal interface material. The highest-power parts drop the metal lid entirely and press the cooler against the bare die, deleting one resistance from the chain.
• The cold plate: a copper block laced with hair-width microchannels, bolted where the air heatsink used to sit. Water-glycol coolant enters as warm as 45°C, gains under 10°C crossing each plate, and can return from the rack near 65°C. The fluid is sealed plumbing; it never touches silicon.
• Rack manifolds and quick-disconnects: vertical pipes feed every tray through zero-drip couplings standardized by the Open Compute Project, so a technician can pull a server without raining coolant on the rack below.
• The coolant distribution unit (CDU): a pump-and-heat-exchanger station, in-rack, sidecar, or row-scale, that hands heat from the clean chip loop to dirtier facility water without ever mixing the two.
• Heat rejection: facility water reaches the sky through dry coolers (giant radiators), evaporative towers (cheap, but they drink water), or chillers (refrigeration that costs power). Because the chip loop returns warm, many liquid-cooled sites run dry coolers year-round.

Source: NVIDIA GB200 NVL72 cooling specifications; OCP UQD specifications; Uptime Institute Global Data Center Survey 2025; full chain notes in docs/research/ai_datacenter_cooling_explainer_2026.md (July 2026)

Air gives way to liquid at high density

Air is a weak carrier: a litre of water holds thousands of times more heat than a litre of air. Past roughly 50 kW per rack, no fan wall can move enough air through the fins, so direct-to-chip liquid cooling takes over and moves heat through cold plates before it ever spreads into the room.

The NVL72 rack at ~120 kW made liquid mandatory rather than optional, and immersion, where whole servers sink into dielectric fluid, supports still higher densities. Each step changes service workflows, vendor choices, and building design.

3D Schematic: Fluid absorbs junction heat crossing the cold plate, returning through vertical manifolds to the facilities loop.

Density decides the technology

Hopper racks at ~40 kW could still breathe air. Blackwell at ~120 kW cannot. The Kyber-class racks NVIDIA has sketched for the Rubin era run toward 600 kW to 1 MW, and at that density single-phase cold plates run into their own limits: pushing coolant through ever-finer channels costs pressure faster than it buys heat transfer.

Past that point the frontier moves in two directions. Two-phase cooling lets the fluid boil on the plate, spending latent heat instead of flow rate. And cooling moves into the silicon itself: Microsoft and Corintis showed microchannels etched into the back of the die in September 2025, removing heat about three times better than an external plate, and Cerebras already pumps water directly at a 23 kW wafer. For 3D-stacked parts that bury logic under HBM, there is no lid path left, so embedded cooling looks less like an option and more like a requirement of the next packaging generation.

Source: NVIDIA GTC 2025 Kyber disclosures; Microsoft microfluidics announcement, September 2025; Cerebras Semi Doped, 2025

800 volts and liquid cooling are one decision

Delivering 120 kW at 48 V takes about 2,500 amperes; at 800 V it takes about 150. Since resistive loss scales with the square of current, the 800 V DC transition exists to feed denser racks through less copper, and the busbar volume it frees becomes room for coolant manifolds.

The coupling runs deeper than shared space. Power conversion loses 2 to 4% of everything it touches, and in a 120 kW rack that loss is kilowatts of heat inside the power shelves themselves, so the PSUs get cold plates too. This is why the vendors that matter here, Vertiv, Delta, Schneider with Motivair, sell power delivery and thermal management as one product line: at these densities they are one engineering problem.

The parts that melt first

The GPU logic die is the toughest component in the package, tolerating junction temperatures near 100°C. HBM needs to stay meaningfully cooler, roughly 85 to 95°C, because DRAM leaks charge faster as it heats, and a 12-high stack insulates its own inner dies. Memory vendors thread dummy copper vias through the stack as heat pipes, and cold plates now extend dedicated cool zones over the memory.

Silicon photonics is the most fragile passenger. Laser wavelengths drift with temperature at roughly 0.1 nm per °C, which is why co-packaged optics, placing optical engines millimeters from a kilowatt die, leans on tuning heaters that burn power to hold temperature, or keeps the laser off-package in a cool, replaceable front-panel module.

The package itself fights back. Silicon, organic substrate, and copper cold plate all expand at different rates, so every thermal cycle shears the micro-bumps between them. Operators now ramp coolant temperatures gently on warm-up and cool-down for the same reason you do not pour boiling water into a cold glass.

Cooling failure is a cluster failure

An air-cooled server that loses a fan drifts warmer over minutes. A kilowatt-class GPU that loses coolant flow spikes to throttle temperatures in seconds, so firmware watches flow switches and executes emergency shutdown rather than letting the silicon cook.

That failure profile shapes the operating model: redundant pumps with sub-second failover, rope leak sensors under every manifold, automatic isolation valves, and a coolant-chemistry discipline (pH, filtration, biocides) closer to running a power plant than racking servers. A leak above a $40,000 GPU is a hardware loss; a stalled loop under a training run is lost model time.

Water, climate, and permitting enter the stack

Cooling ties AI infrastructure to local climate, water availability, environmental permits, and public tolerance. A site that works thermally in one region may be politically or physically hard in another.

This is why data-center geography is not only about cheap power. It is also about where heat can be rejected reliably, and warm-water liquid loops are quietly redrawing that map: a site that rejects heat through dry coolers needs almost no water where an evaporative design consumed litres per kilowatt-hour.

Thermals shape chip choice

A hotter accelerator may be worth it if the rack and facility are built for it. The same chip can be a bad fit in a legacy building with lower density assumptions.

The buyer is not choosing a chip in isolation. The buyer is choosing a chip plus a cooling architecture plus an operating model.

The durable edge is thermal headroom

Sites with liquid-ready distribution, serviceable rack layouts, and room for denser future systems can absorb new chip generations faster.

That headroom is a strategic asset because the hardware clock is faster than the building clock.