Search AI Chips News Today

BUYER’S GUIDE

AI Chip Buyer’s Guide

Compare accelerators for the workloads you plan to run.

PCIe AI accelerator expansion card and heatsink assembly on test bench

Training or inference

Start with the workload. Training typically needs higher compute and memory bandwidth to process large datasets and update models. Inference often prioritises efficiency, predictable latency and the ability to handle many concurrent requests.

Memory requirements

Model size, batch size and input data all drive memory needs. Aim for enough accelerator memory to keep the model and working data in memory, with headroom for your expected workload. Also consider memory bandwidth, which affects how quickly data can move to the compute cores.

Software support

A strong software ecosystem makes it easier to build, optimise and deploy models. Check for mature drivers, widely used frameworks and a healthy set of tools for performance analysis and deployment in your target environment.

Power and cooling

Accelerators can draw substantial power and add heat to your data centre. Make sure your racks, power delivery and cooling can handle the sustained load, with enough margin for future growth and higher utilisation.

CoreWeave Trains DeepSeek-V3 in 2.02 Minutes on 8,192 GB300 GPUs

Frequently Asked Questions: Selecting AI Accelerators

How much HBM memory do I need to serve a 70B or 405B parameter LLM?

In 16-bit precision (BF16/FP16), model weights alone consume ~2 bytes per parameter (~140 GB for 70B; ~810 GB for 405B), plus 20–40% additional HBM for the KV-cache context buffer and runtime activations. Using FP8 (1 byte/param) or NVFP4/INT4 (0.5 bytes/param) cuts weight memory by 50% to 75%, allowing a 70B model to run on a single 80 GB–141 GB GPU and a 405B model to fit within a single 8-GPU node or dual-socket 256 GB/288 GB HBM3E setup.

When does an AI cluster require direct-to-chip liquid cooling instead of air cooling?

Air cooling with hot/cold aisle containment typically reaches its practical acoustic and thermal ceiling around 35 kW to 45 kW per rack. When deploying 8-GPU baseboards drawing 10 kW–14 kW per chassis or 72-GPU rack-scale systems (120 kW to 227 kW per rack), direct-to-chip cold-plate liquid cooling paired with a Coolant Distribution Unit (CDU) is mandatory.

Should enterprise teams rent cloud AI chips or buy on-premises GPU servers?

Bursty training runs and elastic inference workloads often achieve lower total cost on cloud computing AI chips (such as AWS Trainium2, Google Cloud TPU v6e/Ironwood or Microsoft Azure Maia and GPU instances). Sustained 24/7 production inference with predictable utilization above 65–70% over a 36-month amortization cycle often favors dedicated colocation or on-premises hardware.

Why does memory bandwidth matter more than peak TFLOPS for LLM token decoding?

Autoregressive token generation reads the entire active model weight matrix from HBM into SRAM for every generated step. At small-to-moderate batch sizes, arithmetic intensity is low, making per-user token speed directly proportional to HBM bandwidth (TB/s) rather than peak matrix TFLOPS.

More from AI Chips News Today