NVIDIA’s NVFP4 KV cache quantization reduces the key-value cache memory footprint by up to 50% and shifts native 16-bit KV storage to 4-bit, a change that can effectively double context budgets for inference while holding accuracy loss below one percent, according to NVIDIA’s developer guidance on NVFP4 KV cache. The technique arrives alongside Blackwell’s native FP4 Tensor Cores and follows Hopper’s native FP8 support, while vLLM has integrated FlashInfer kernels that deliver 38% more peak throughput and 13% lower latency.
How NVFP4 KV cache reduces memory and compute overhead
In a transformer model, the KV cache stores key and value tensors once instead of recomputing them for each generated token. Cache-hit rate determines whether the cache delivers its intended compute savings. TensorRT Model Optimizer quantizes the KV cache from native 16-bit to 4-bit. The current NVFP4 implementation dequantizes values to FP8 before attention math, while new token K/V vectors are quantized to NVFP4 before appending to the cache. The Model Optimizer quantize API supports both post-training quantization and quantization-aware training, and enabling NVFP4 KV cache requires only changing the quantization configuration. Weights can also move to NVFP4 via NVFP4_DEFAULT_CFG for 4-bit math. FP4 compresses weights to a quarter of FP16 size, or 0.5 bytes per parameter, but sensitive layers and attention often remain FP8 while most weights shift FP4. FP4 gains disappear on hardware lacking native Tensor Cores.
Key figures from the available reports:
| Metric | Value | Context |
|---|---|---|
| KV cache memory footprint reduction | 50% | NVFP4 versus native 16-bit KV cache |
| Native KV cache precision | 16-bit | Before quantization by TensorRT Model Optimizer |
| Memory per parameter for FP4 | 0.5 bytes | Weight compression versus FP16 |
| Peak throughput gain | 38% | vLLM FlashInfer FP8 attention and FP4 GEMM kernel optimization |
| Minimum latency reduction | 13% | Same kernel optimization |
| Accuracy loss under NVFP4 KV cache | Below 1% | Reported by NVIDIA |
Workload impact on TTFT, token throughput, and accuracy
Prefill ingests the full input sequence, while decode generates tokens one at a time. Long context and high batch sizes push harder against KV cache capacity, so halving the KV memory footprint can stretch context budgets. NVIDIA’s documentation states NVFP4 KV cache doubles content budget over a standard FP8 KV cache. However, the full performance impact on time-to-first-token and token throughput at long context and high batch sizes has not been quantified in the published reports. The current implementation dequantizes cached values to FP8 before the attention operation, so the attention math itself remains FP8 even though storage is NVFP4. This hybrid design preserves compute compatibility while trading lower cache memory. The vLLM integration with FlashInfer for FP8 attention and FP4 GEMM kernels delivered a kernel-level 38% peak throughput gain and 13% lower latency, but those figures are separate from the NVFP4 KV cache quantization. For infrastructure teams, cost per token matters more than raw GPU utilization, and evaluating the AI accelerator software stack should include quantization support, kernel maturity, and long-context behavior.
Independent benchmarks and the open InferenceMAX effort
SemiAnalysis has released InferenceMAX, an open-source benchmarking effort that runs nightly on hundreds of chips automatically and exposes results through a public live dashboard. InferenceMAX v1 covers GB200 NVL72, B200, MI355X, H200, MI325X, H100, and MI300X. AMD and Nvidia each lead on different workloads and interactivity levels. ROCm issues now occur far less frequently than at the end of 2024. Benchmarks fixed in time go stale as software improves within days, which is why continuous automated runs matter. Google TPU and AWS Trainium backends are scheduled to be added within the next two months. Peter Hoeschele, VP of Infrastructure and Industrial Compute at OpenAI Stargate, said:
> As we build systems at unprecedented scale, it's critical for the ML community to have open, transparent benchmarks that reflect how inference really performs across hardware and software. InferenceMAX's head-to-head benchmarks cut through the noise and provide a living picture of token throughput, performance per dollar, and tokens per Megawatt. This kind of open source effort strengthens the entire ecosystem and helps everyone, from researchers to operators of frontier datacenters, make smarter decisions.
These open results complement standardized submissions such as MLCommons (MLPerf Benchmarks), and the AI chip benchmarks hub tracks both continuous and fixed benchmarking approaches.
What remains unannounced and what to verify before procurement
Several gaps remain. The full performance impact of NVFP4 KV cache on TTFT and token throughput at long context and high batch sizes is not quantified in the available reports. Independent third-party benchmarks of NVFP4 KV cache have not yet been cited. InferenceMAX expansion to Google TPU and AWS Trainium backends is still pending, and its v1 is described as far from perfect, with future workload and model coverage refinements planned. Ongoing kernel optimization gains beyond the documented 38% throughput and 13% latency figures remain open.
Infrastructure teams should verify the following before committing to low-precision serving:
- Request independent long-context, high-batch TTFT and token throughput numbers for NVFP4 KV cache on your target hardware.
- Verify accuracy loss below one percent on your own model and dataset rather than relying only on vendor reports.
- Measure cost per token and tokens per Megawatt under your actual request mix, including cache-hit patterns.
- Confirm native FP4 or FP8 Tensor Core support and serving stack kernel maturity before procurement.
A careful evaluation should use the AI Chip Buyer’s Guide and compare training versus inference accelerator trade-offs described in comparing training vs inference AI accelerators before making infrastructure decisions.