Software support: evaluating AI compiler stacks, drivers and runtimes
A strong software ecosystem makes it easier to build, optimise and deploy models without rewriting custom low-level assembly kernels from scratch. In enterprise hardware evaluations, an accelerator with impressive theoretical datasheet FLOPS frequently underperforms in production if its graph compiler, collective communications library or serving runtime lacks fused kernels for your exact model architecture.
1. The four layers of a production AI software stack
Before committing capital to merchant GPUs or cloud ASICs, benchmark the target hardware across all four software layers using your actual model checkpoints:
- Layer 1 — Kernel & math libraries: High-efficiency transformers depend on fused attention kernels (FlashAttention-2/3, PagedAttention), quantized matrix multiplication (CUTLASS, hipBLASLt, Gemmlowp) and fused RoPE/RMSNorm/SwiGLU operators that minimize HBM round-trips.
- Layer 2 — Graph compilers & kernel DSLs: Modern teams write hardware-portable kernels in OpenAI Triton, PyTorch 2 (`torch.compile` / TorchInductor) or JAX / OpenXLA. Verify whether the vendor’s Triton and XLA backends achieve within 85–95% of hand-tuned vendor assembly on your target precision (FP8, FP6 or FP4).
- Layer 3 — Production inference engines: Confirm upstream day-zero integration with vLLM, SGLang, TensorRT-LLM or ONNX Runtime, including support for continuous batching, speculative decoding, prefix caching and disaggregated prefill/decode routing.
- Layer 4 — Collective communications & observability: Multi-GPU scaling relies on deterministic collective libraries (NCCL, RCCL, or XLA collectives) and fleet telemetry exporters (DCGM, AMD SMI, Prometheus) that detect silent data corruption (SDC) and thermal throttling.
| Hardware Platform | Primary Compute Stack | Compiler & Kernel Ecosystem | Supported Serving Engines |
|---|---|---|---|
| NVIDIA Blackwell / Hopper / Rubin | CUDA, cuDNN, cuBLAS, CUTLASS, NCCL | OpenAI Triton, PyTorch 2, TensorRT | TensorRT-LLM, vLLM, SGLang, Dynamo |
| AMD Instinct (MI300X / MI325X / MI355X) | ROCm, hipBLASLt, Composable Kernel, RCCL | OpenAI Triton (ROCm backend), PyTorch 2 | vLLM, SGLang, ONNX Runtime, llama.cpp |
| Google Cloud TPU (v6e Trillium / Ironwood) | OpenXLA, Pathways, ICI Collectives | JAX, Pallas (Triton-like kernel DSL), PyTorch/XLA | JetStream, vLLM-TPU, MaxText |
| AWS Trainium2 / Inferentia2 | AWS Neuron SDK, NeuronLink Collectives | NeuronX Compiler, NKI (Neuron Kernel Interface) | vLLM (Neuron backend), Text Generation Inference (TGI) |
2. Red flags to catch during proof-of-concept testing
- Benchmark-only kernel paths: Ask whether measured performance relies on open, upstream vLLM/PyTorch branches or proprietary closed binaries tailored only to static MLPerf sequence lengths.
- Container & Kubernetes operator maturity: Verify GPU/XPU device plugins, Multi-Instance GPU (MIG) or spatial partitioning support, and RDMA SR-IOV compatibility on your Linux kernel version.
See how software stacks perform on real workloads in our InferenceMAX v1 AMD vs NVIDIA benchmark report and Cloud AI chips comparison, or return to the AI Chip Buyer’s Guide.