Search AI Chips News Today

BUYER’S GUIDE

Choosing the right interconnect for AI workloads

High-density optical fiber interconnect cabling in an AI data centre rack

Once a foundation model outgrows the memory of a single accelerator or a training cluster scales past a single server chassis, interconnect bandwidth and tail latency determine whether expensive tensor cores stay productive or sit idle waiting on collective communications. Modern AI cluster networking is divided into two distinct domains: scale-up (intra-node / intra-rack) and scale-out (inter-rack / cluster-wide).

1. Scale-up fabrics: byte-addressable chip-to-chip domains

Scale-up links carry latency-sensitive Tensor Parallelism (TP) and Expert Parallelism (EP) traffic, where accelerators exchange activations after every single attention and feed-forward layer:

  • PCIe Gen5 / Gen6 (64 GB/s to 128 GB/s bidirectional): Sufficient for independent single-GPU inference, data-parallel fine-tuning of smaller models, or pipeline-parallel stages with minimal cross-chip chatter.
  • 8-GPU Baseboard Fabrics (NVLink 4/5, AMD Infinity Fabric): Connects 8 SXM or OAM accelerators inside a single server node at 600 GB/s to 1,800 GB/s per GPU, allowing an 8-GPU chassis to act as a unified 1.4 TB to 2.3 TB HBM memory pool.
  • Rack-Scale Switch Fabrics (NVLink 5/6 NVL72, UALink, Google ICI): Extends high-bandwidth memory semantics across 36 to 72+ GPUs within a liquid-cooled rack (reaching 3.6 TB/s per GPU on NVLink 6) via copper backplanes or optical circuit switches.

2. Scale-out fabrics: RDMA across hundreds or thousands of racks

Scale-out networks connect multiple nodes or racks using Remote Direct Memory Access (RDMA) at 400G, 800G and 1.6T OSFP optical speeds (typically provisioning 1 dedicated backend NIC per GPU in a rail-optimized leaf-spine or dragonfly topology):

  • InfiniBand (NDR 400G / XDR 800G): Purpose-built credit-based lossless fabric with adaptive routing and SHArP in-network aggregation, minimizing tail latency on synchronous AllReduce gradients.
  • RoCEv2 & Ultra Ethernet Consortium (UEC): High-radix Ethernet switches (51.2 Tbps and 102.4 Tbps) paired with SmartNICs/DPUs that implement packet spraying, selective retransmission and congestion telemetry to deliver InfiniBand-class collective efficiency over multi-vendor Ethernet ecosystems.
Interconnect Tier Bandwidth per Accelerator Primary Parallelism Mode Physical Medium & Reach
Scale-Up (Intra-Node / Rack) 600 GB/s – 3,600 GB/s Tensor Parallelism (TP) & MoE All-to-All PCB traces, Twinax copper backplane (<3m)
Scale-Out Backend (East-West) 400 Gbps – 1,600 Gbps (50–200 GB/s) Data Parallelism (FSDP/ZeRO) & Pipeline (PP) DAC/AOC & single-mode DR8/2xFR4 optics (10m–2km)
Frontend / Storage Network 100 Gbps – 400 Gbps per host Checkpoint I/O, dataset streaming & API ingress Standard data centre leaf-spine Ethernet

Back to the AI Chip Buyer’s Guide, review Power and cooling for AI servers, or explore our Data Centres hub.