Search AI Chips News Today

Benchmarks

InferenceMAX v1: AMD vs NVIDIA Inference Results by Workload

SemiAnalysis InferenceMAX v1 measures latency, throughput, and cost per token across AMD and NVIDIA. See workload winners, Blackwell gains, and software gaps.

NVIDIA H100 (极客湾Geekerwan) 002. Photo: 极客湾Geekerwan (CC BY 3.0, Wikimedia Commons).

SemiAnalysis has launched InferenceMAX v1, an open-source inference hardware evaluation methodology, and NVIDIA says its GPUs deliver the highest inference performance across all workloads under that benchmark. A separate six-month validation by SemiAnalysis finds that AMD versus NVIDIA inference performance per TCO depends on workload: chat, document processing, and reasoning show different winners. The methodology publishes daily CI benchmark sweeps across SGLang, TensorRT-LLM, and vLLM, and reports latency, throughput, batch sizes, and input/output ratios, as detailed in the SemiAnalysis AMD vs NVIDIA inference benchmark and the NVIDIA Blackwell InferenceMAX v1 technical blog.

What does InferenceMAX v1 actually measure for inference servers?

InferenceMAX v1 measures latency, throughput, batch sizes, and input/output ratios across SGLang, TensorRT-LLM, and vLLM, using daily CI sweeps. Test scenarios use input sequence length/output sequence length (ISL/OSL) combinations of 1K/1K for chat, 8K/1K for summarization, and 1K/8K for deep reasoning. The methodology tests variable sequence lengths of 80-100% of ISL/OSL combinations to reflect real-world deployments. Models tested include DeepSeek-R1, a 671B Mixture-of-Experts with 37B active parameters in FP8 and NVFP4; gpt-oss-120b, a 117B MoE with 5.1B active parameters in FP8 and MXFP4; and Llama 3.3 70B, a dense 70B model in FP8 and NVFP4. Concurrency is currently limited to 4-64. For broader context on inference metrics, see the AI chip benchmarks hub and MLCommons (MLPerf Benchmarks) for industry-standard ML benchmark references.

Where does AMD or NVIDIA win on performance per dollar for inference?

For owned GPUs, NVIDIA or AMD leads perf/$ depending on workload. For sub-six-month Neocloud rentals, NVIDIA always wins perf/$ due to scarce AMD supply. MI300X rental rates are elevated because few AMD Neoclouds exist, while hundreds of Neoclouds offer H100 and H200, creating competitive rental pricing. A single-node H200 or MI325X perf/$ advantage depends on workload and latency. SemiAnalysis spent six months validating AMD versus NVIDIA inference performance per TCO and found results varied by workload: chat, document processing, and reasoning showed different winners. MI325X mass shipments began Q2 2025, one quarter after HGX B200 shipments, and MI355X will ship late 2025, two quarters after B200 shipments start. B200 dominates currently deployable workloads; MI325 and H200 are far behind.

Why does NVIDIA report a 15x Blackwell gain over Hopper under InferenceMAX?

NVIDIA attributes Blackwell's 15x performance gain over Hopper to NVFP4, fifth-gen NVLink, TensorRT-LLM, and Dynamo co-design. Blackwell features fifth-gen Tensor Cores with native FP4 acceleration and HBM3e memory, delivering order-of-magnitude compute-per-watt and memory bandwidth gains over Hopper. NVLink and NVLink Switch provide 1,800 GB/s bidirectional bandwidth, avoiding PCIe bottlenecks. On October 9, a TensorRT-LLM release added EP and DEP mappings, raising max throughput up to 5x. In DEP2, attention runs on one GPU while MoE experts route across two GPUs, 64 experts per GPU. Speculative decoding enabled for gpt-oss-120b using gpt-oss-120b-Eagle3-v2 triples per-GPU throughput at 100 TPS/user, from 10K to 30K tokens/s. At 400 TPS/user, cost per million tokens remains low at $0.12.

What software and CI gaps should engineers monitor before choosing AMD ROCm or NVIDIA CUDA?

SemiAnalysis still sees much room for AMD inference software improvement; ROCm SGLang CI coverage is under 10% parity with NVIDIA, and models score worse on ROCm than CUDA due to missing CI and numeric accuracy kernels. AMD added some Continuous Integration automated tests to improve developer experience, but SemiAnalysis notes that AMD added environment variables despite advice to remove them, worsening configuration complexity. Too many serving framework flags create combinatorial explosion, hindering comprehensive benchmarking. NVIDIA TRT-LLM developer experience improved but still trails vLLM and SGLang; TRT-LLM needs full DeepSeek support and prebuilt TRT-LLM-serve container images. SemiAnalysis has collaborated with AMD since Q3 2024, and AMD internal R&D cluster spending last quarter was ~$13M, while AMD stock buybacks last quarter were $749 million. Lack of R&D clusters drives AMD's weaker developer experience and software lag.

What should infrastructure buyers scrutinize before purchasing or renting inference servers?

Buyers should scrutinize workload-specific winners, rental supply, and software maturity, as these determine real cost per token more than peak throughput alone. Practical considerations include:

  • Check workload and latency: chat, document processing, and reasoning show different winners; single-node H200 or MI325X perf/$ advantage depends on workload and latency.
  • For owned deployments, compare perf/$ across your actual ISL/OSL ratios; InferenceMAX uses variable sequence lengths of 80-100% of ISL/OSL combinations.
  • For short-term rental under six months, assume NVIDIA wins perf/$ because hundreds of Neoclouds offer H100/H200, while AMD MI300X rental rates are elevated from scarce supply.
  • Verify framework support: TensorRT-LLM needs full DeepSeek support and prebuilt serve containers; ROCm SGLang CI parity is below 10%, leading to worse model scores on ROCm than CUDA. See evaluating the AI accelerator software stack for a deeper checklist.
  • Look beyond peak throughput: evaluate concurrency (currently 4-64), cost per million tokens at realistic interactivity (e.g., $0.12 at 400 TPS/user), and memory/bandwidth features like NVLink 1,800 GB/s.

For a broader framework on training versus inference accelerators, see the AI Chip Buyer’s Guide. What remains unannounced: full support for DeepSeek in TRT-LLM is still needed, prebuilt TRT-LLM-serve container images are not yet available, B200 and GB200 software is not fully fleshed out, FP8 DeepSeek V3 is not fully working on TRT-LLM, vLLM, or SGLang, and future Dynamo integrations are planned but not yet covered in the dashboard.