Choosing the right memory for AI workloads
Model parameter count, concurrent batch size and context sequence length all drive accelerator memory requirements. In production deployments, running out of package High Bandwidth Memory (HBM) forces tensor splitting across additional GPUs or offloading to slower host DDR5/LPDDR5X memory over PCIe/CXL — immediately increasing tail latency. Aim to keep all active model weights and working KV-cache inside local HBM with 15% to 20% headroom for runtime activation buffers.
1. Step-by-step formula for sizing LLM memory capacity
Total accelerator memory required for inference is the sum of three components: M_total = M_weights + M_kv_cache + M_overhead.
- Model weights (
M_weights): Multiply total parameters by bytes per parameter:- BF16 / FP16 (2 bytes/param): A 70B model requires ~140 GB; a 405B model requires ~810 GB.
- FP8 (1 byte/param): A 70B model requires ~70 GB; a 405B model requires ~405 GB.
- NVFP4 / INT4 (0.5 bytes/param): A 70B model requires ~35–38 GB; a 671B DeepSeek-R1 Mixture-of-Experts model requires ~336 GB.
- KV-cache memory (
M_kv_cache): Scales linearly with batch size (B), sequence length (S), transformer layers (L), KV attention heads (H_kv) and head dimension (D):2 × B × S × L × H_kv × D × bytes_per_element. As shown in our NVFP4 KV-cache memory analysis, quantizing the KV-cache from BF16 to FP8 or 4-bit cuts long-context memory consumption by 50% to 75%. - Runtime & graph overhead (
M_overhead): Reserve 8 GB to 16 GB per GPU for CUDA/ROCm context, NCCL/RCCL communication ring buffers and CUDA Graph capture pools.
| Memory Technology | Interface Width (per Stack) | Typical Stack Capacity | Bandwidth per Accelerator (8 Stacks) | Target AI Workloads |
|---|---|---|---|---|
| GDDR6 / GDDR7 | 32-bit per chip (256–512-bit bus) | 32 GB – 96 GB card total | 0.8 TB/s – 1.8 TB/s | Small models (7B–32B), vision, embeddings & cost-optimized edge inference |
| HBM3E (8-High / 12-High) | 1,024-bit per stack | 24 GB – 36 GB per stack (192–288 GB total) | 4.8 TB/s – 8.0 TB/s | Frontier 70B–405B LLM training & low-latency FP8/FP4 production serving |
| HBM4 (12-High / 16-High) | 2,048-bit per stack | 36 GB – 64 GB per stack (288–512+ GB total) | 10+ TB/s to 22+ TB/s | Next-gen reasoning models, trillion-parameter MoE & long-context agents |
2. Why HBM4’s 2,048-bit interface changes accelerator design
As detailed in our SK hynix HBM4 architectural breakdown, JEDEC’s HBM4 standard doubles the per-stack data bus from 1,024 bits to 2,048 bits while moving the base die onto an advanced foundry logic process. That allows a single stack to exceed 2.0 TB/s while improving memory subsystem energy efficiency by up to 40% compared with HBM3E.
3. Procurement checklist for memory sizing
- Avoid unnecessary tensor-parallel hops: If a model fits inside 256 GB–288 GB on fewer high-capacity GPUs, you avoid splitting layers across slow inter-node links.
- Verify ECC and RAS telemetry: At cluster scale, multi-bit HBM errors cause training restarts; ensure hardware supports On-Die ECC and Directed Refresh Management.
Read our latest memory and processor coverage on the Chipmakers hub, continue to Evaluating AI accelerator software stacks, or return to the AI Chip Buyer’s Guide.