Explorer · topology

Fat-tree (Clos) interconnect

A k-ary fat-tree stacks three switching tiers — leaf, aggregation, spine — with bandwidth doubling at every level toward the root. The result: any host can talk to any other at full line rate, regardless of where they sit. NVIDIA's NVL72 fabric, AWS, Azure, and most cloud GPU clusters all run a Clos variant. Drag to rotate; the cyan path traces the worst-case 5-hop route between two pods.

4 pods · 4 spines · 16 switches total

fat-tree (Clos)

20 switches

4 pods · 4 spines · capacity 16 hosts

Drag to rotate · scroll to zoom

Tiers

■ spine

■ aggregation

■ leaf

Topology

Fat-tree (Clos)

Three tiers · bandwidth doubles per level toward root

Diameter

5 hops

Worst case: leaf → agg → spine → agg → leaf across pods

Bisection

Non-blocking

Any leaf-to-leaf permutation completes at full line rate

Why a fat-tree?

The job is non-blocking bisection — letting half the network talk to the other half simultaneously without contention. A flat switch fabric can do this but requires a single switch the size of the data centre. A fat-tree gets the same property by stacking small switches into tiers, doubling aggregate bandwidth at every level. Bisection bandwidth scales with the network, not as a fixed root link.

The version most production fabrics actually run is a folded Clos — three (sometimes five) levels of small radix switches arranged so that every leaf has a non-blocking path to every other leaf through some combination of upper-tier switches. That combinatorial flexibility is what ECMP routing exploits to spread flows across thousands of paths.

The 5-hop worst case

For two leaves in the same pod, the path is just three hops: leaf → agg → leaf. For two leaves in different pods, the traffic has to climb to the spine and come back down:

Source leaf → aggregation switch in source pod.
Aggregation → spine switch (any one in the agg's spine group).
Spine → aggregation switch in destination pod.
Aggregation → destination leaf.

Five switching hops, but every hop is single-tier and engineered for full line rate. The latency is dominated by serialisation and queueing, not switching.

Where it shows up

NVIDIA's NVL72 rack is an InfiniBand or NVLink-Switch Clos at the rack scale; cluster-wide GPU fabrics extend the same pattern over RoCE or InfiniBand. AWS, Azure, GCP, and Meta's in-house networks all run Clos at multiple tiers. Compare with the 3D torus explorer (TPU pods) and the dragonfly explorer (HPC supercomputers).

Rule of thumb: if the workload mix is unpredictable (cloud, multi-tenant), Clos wins because non-blocking bisection makes any permutation safe. If the workload is one giant all-reduce (TPU training), torus wins because directly-wired neighbours beat tier-walking.

Power Silicon Infra Models Apps

Back to ideas