NVIDIA's first MLPerf Inference results on the Blackwell Ultra architecture recorded 5,842 tokens/second/GPU offline throughput for the 671-billion-parameter DeepSeek-R1 mixture-of-experts reasoning model on GB300 NVL72, using the company's NVFP4 4-bit floating-point format. The debut arrives six months after Blackwell's v5.0 appearance, and NVIDIA retains per-GPU records on all other MLPerf inference benchmarks. These results place NVFP4 at the center of Blackwell Ultra's inference strategy, with peak dense NVFP4 compute of 15 petaFLOPS—3x the FP8 throughput on the same GPUs.
NVFP4 format and Blackwell Ultra architecture
NVFP4 is a 4-bit floating-point format with 1 sign bit, 2 exponent bits, and 1 mantissa bit, yielding a value range from approximately -6 to 6. Instead of a single scale per tensor, NVFP4 applies an E4M3 FP8 scaling factor per 16-value micro-block plus a second-level FP32 scalar per tensor. NVIDIA reports an average mean squared error of 0.08 for E4M3 scaling, lower than the E8M0 alternative. Blackwell's fifth-generation Tensor Cores handle microscaled FP4 data and 4-bit matrix operations natively, so no explicit dequantization step is needed before the matrix multiply. From a hardware perspective, Blackwell Ultra adds 1.5x peak NVFP4 AI compute compared with Blackwell, doubles attention-layer compute, and offers 1.5x higher HBM3e capacity. The format's 4-bit storage can use up to 4x less memory than FP16, which shifts capacity-constrained deployments toward larger batch sizes or longer context windows. More details are in NVIDIA's NVFP4 format introduction.
MLPerf throughput on DeepSeek-R1, Llama, and Whisper
The GB300 NVL72 submissions show a wide spread across workloads. DeepSeek-R1 offline reaches 5,842 tokens/second/GPU, while the server scenario drops to 2,907 tokens/second/GPU. For Llama 3.1 405B, offline throughput is 224 tokens/second/GPU, server is 170, and interactive is 138. Llama 3.1 8B is far more compute-light at 18,370 offline, 16,099 server, and 15,284 interactive tokens/second/GPU. Whisper offline hits 5,667 tokens/second/GPU, a workload also seeing nearly 5 million HuggingFace downloads in one month. NVIDIA submitted NVFP4 results for DeepSeek-R1, Llama 3.1 8B and 405B, and Llama 2 70B; the MLPerf closed division requires meeting accuracy thresholds, so the NVFP4 quantization did not invalidate submissions. NVIDIA notes that moving from FP8 to NVFP4 improves DeepSeek-R1 token throughput at a given interactivity level. The comparative Hopper numbers—1,253 tokens/sec/GPU offline and 556 tokens/sec/GPU server for DeepSeek-R1 on DGX H200—were not verified by MLCommons Association; NVIDIA's Blackwell Ultra delivered about 5x Hopper throughput per GPU on those unverified results. Since per-chip performance is not a primary metric of the MLPerf Inference benchmark, the reported per-GPU records reflect single-node scaling rather than a full-system ranking. See NVIDIA's Blackwell Ultra MLPerf results.
NVFP4 software stack and training implications
The same 4-bit format is also being used for training. 512 Blackwell Ultra GPUs pre-trained Llama 3.1 405B in 64.6 minutes using NVFP4, which NVIDIA says is 1.9x faster than 512 Blackwell GPUs using FP8. For inference, TensorRT-LLM, vLLM, and SGLang support running models in NVFP4. Black Forest Labs reported up to a 6.3x speedup on a single B200 using NVFP4 along with CUDA Graphs, torch.compile, and TeaCache. Robin Rombach, co-founder and CEO of Black Forest Labs, said: "By layering optimizations like CUDA Graphs, torch.compile, NVFP4 precision, and TeaCache, we achieve up to a 6.3x speedup on a single B200—dramatically reducing latency and enabling more efficient production deployment." Red Hat uses NVFP4 to get near-baseline accuracy within tight memory budgets. The format's 4-bit memory footprint is particularly relevant for serving MoE models like DeepSeek-R1, where expert weights dominate capacity.
Practical deployment takeaways and unverified specifications
- Blackwell Ultra's 15 petaFLOPS peak dense NVFP4 is 3x FP8 on the same GPU; workload throughput scales only with operators that saturate Tensor Core FP4 paths.
- The 1.5x attention-layer compute and 1.5x HBM3e capacity versus Blackwell support longer KV caches and larger expert counts without falling back to host memory.
- NVFP4's E4M3 per-16-value micro-block scaling produced average MSE 0.08; verify outlier layers and long-sequence accuracy, especially for models whose activations exceed the roughly -6 to 6 value range.
- Rubin's announced 35 petaFLOPS NVFP4 training and 50 petaFLOPS NVFP4 inference represent 3.5x and 5x Blackwell, but those platforms are not yet available for purchase.
- For production choices, consult the AI Chip Buyer’s Guide and evaluating the AI accelerator software stack. Benchmark context across AI chip benchmarks hub and comparing training vs inference AI accelerators.
The Hopper-based DeepSeek-R1 comparisons were not verified by MLCommons Association, so the 5x per-GPU uplift should be treated as a vendor-reported directional figure. The MLPerf Inference benchmark does not treat per-chip performance as a primary metric, which means system-level rankings can shift with interconnect and topology. Rubin's NVFP4 compute figures are forward-looking; the platform is not available, and real workload performance will depend on software maturity. For official submission rules and accuracy thresholds, refer to MLCommons (MLPerf Benchmarks). The NVFP4 training speedup of 1.9x on 512 Blackwell GPUs is also a vendor-provided result, not an MLPerf training benchmark submission; evaluate your own data distribution before assuming equivalent convergence. Finally, while NVIDIA's 3 Ways NVFP4 Accelerates AI Training and Inference describes the format in production, independent validation across model families and quantization-sensitive tasks is still sparse.