The Real Numbers Behind Low-Precision Training: A Practical GEMM-Level Guide for Transformer Engineers

If you have been building AI infrastructure long enough, you have learned the hard way that benchmark numbers are not product numbers. The spec sheet says FP8 delivers 2x throughput over BF16. The real training run delivers something different, and the gap is usually not in NVIDIA's favor. A post NVIDIA published this week does something unusual: it explains exactly why the gap exists, which matrix multiplications in a transformer actually benefit from lower precision, and how to measure it yourself before committing to a training campaign that could cost hundreds of thousands of dollars in GPU time.

The post is a technical walkthrough for Transformer Engine's GEMM microbenchmarking tool — a script that derives the 12 specific matrix multiplication shapes in your transformer (Fprop, Dgrad, and Wgrad stages for QKV projection, attention output, MLP up, and MLP down) and measures each one at FP8, MXFP8, and NVFP4 precision on your actual hardware. The concrete benchmarking data is from CodonFM 5B on a B300, and it contains enough surprises to make any ML engineer rethink what "low-precision training" actually means for their workload.

The most important finding is the GEMM shape dependency. NVFP4 delivers a 1.66x speedup over MXFP8 on MLP Down — that is a large, square(ish) matrix multiplication where the tensor core pipeline fills efficiently and quantization overhead amortizes cleanly. The same NVFP4 format delivers only a 1.05x speedup on Attention Output — a 4096×4096 GEMM that is too small to hide the cost of Hadamard transforms, stochastic rounding, 2D block scaling, and per-tensor amax computation. These are not marginal differences. They are the difference between "low precision is faster" and "low precision on this specific operation is barely faster than the alternative."

The blended Fprop speedup — accounting for Wgrad and all non-GEMM overhead — comes in at roughly 1.47x for NVFP4 over BF16 on CodonFM 5B. That is a real speedup. It is also significantly lower than the 2-3x headline numbers that get cited in hardware specs and conference talks. The reason is that every training step includes work that is not a GEMM: normalization, attention computation, dropout, gradient communication, optimizer state updates. When those components are included in the measurement, the aggregate speedup converges toward the slowest bottleneck in the pipeline, which is often not the GEMM.

NVIDIA's autocast versus prequantized comparison is the methodological contribution that most teams miss. Autocast mode is what you actually use in training: dynamic quantization happens at each step, and the measured time includes quantization overhead. Prequantized mode removes that overhead by running the quantized GEMM directly, which tells you the kernel-only performance. The gap between the two tells you whether you are bottlenecked on the tensor core math or on the quantization plumbing. For NVFP4 on B300, the gap is dramatic: 1.98x in autocast versus 3.48x in prequantized. That 1.5x difference is pure quantization overhead, and it means that teams evaluating NVFP4 from autocast-only benchmarks are systematically underestimating what the hardware can do and systematically overestimating the actual training speedup they will see.

The QKV Proj Dgrad asymmetry is the detail that will save engineers from making wrong hardware decisions. During backward pass, the gradient computation for QKV projection swaps the K and N dimensions — transforming a roughly 3:1 aspect ratio matrix (head_dim=4096, hidden=12288 in the forward pass) into a roughly 1:3 aspect ratio in the backward pass. CUDA kernel selection is sensitive to aspect ratio, and the result is that QKV Proj Dgrad runs 33-51% slower than QKV Proj Fprop for FP8 and FP4 formats, versus only about 2% slower for BF16. If you profile Fprop and assume Dgrad is approximately the same speed, your training time estimates will be wrong. If you profile only in BF16 and assume the ratio transfers to FP8, your precision format decision may be wrong.

MXFP8 does not need Hadamard transforms or stochastic rounding, which is why it stays more competitive with NVFP4 in autocast mode than the raw tensor-core specifications would suggest. FP8 DelayedScaling outperforms both FP8 CurrentScaling and MXFP8 in autocast mode on Blackwell (7.80 ms/layer versus 9.15 and 8.98), but FP8 CurrentScaling wins in prequantized mode (6.81 ms versus 8.12). The winner depends entirely on whether you are measuring with or without quantization overhead — and most teams benchmarking autocast in isolation are not getting the full picture.

The silent kernel fallback is the operational surprise that will bite teams first. Transformer Engine can fall back to FP8 or BF16 for layers that do not have FP4 kernel support yet, which produces identical numerical results with no error messages or warnings. You pay the memory cost of FP4 weights — they occupy 4 bits each — but you get FP8 or BF16 performance from those layers. The detection method is GPU memory comparison between runs: if NVFP4 and MXFP8 consume nearly the same memory, your FP4 weights are not being used. NVTE_LOG_LEVEL=1 or NVIDIA Nsight Systems confirms whether FP4 kernels are actually dispatching.

For teams evaluating low-precision training formats: the right workflow is to run the Transformer Engine benchmark on your actual model shapes, batch sizes, and hardware before committing to a precision format. The tool takes a transformer config and derives the exact M×K×N shapes, so there is no excuse for not knowing which GEMMs dominate your step time. If your attention output GEMMs are small — and for most decoder-only models they are — NVFP4 will not deliver the headline speedup. If your MLP Down is large and dominates your step time, the speedup will be real and measurable. The tool tells you which case you are in.

The broader lesson is about precision format economics. Hopper introduced FP8 as the production low-precision format. Blackwell added NVFP4 as the next step down the precision ladder. Each new format has a window where the tooling, kernel coverage, and benchmark suite are immature, and early adopters discover the gap between the hardware spec and their real training speedup. Transformer Engine's open-source benchmarking tool is NVIDIA's answer to that problem: make the measurement infrastructure free and reproducible so teams can answer the question for their specific shapes rather than relying on vendor benchmarks that were almost certainly run on different model architectures, different batch sizes, and different software versions than what you are running.

The actionable engineering takeaway is not "use NVFP4" or "use MXFP8." It is "benchmark your specific workload before choosing a format." The precision format decision cascades into memory footprint, convergence behavior, hardware compatibility, and software maturity — all of which vary by model architecture, training framework version, and hardware generation. The Transformer Engine GEMM benchmarking tool makes that measurement tractable. There is no good reason to make the format decision on anything other than measured results from your own system.

Sources: NVIDIA Developer Blog, NVIDIA Transformer Engine GitHub, GEMM profiling tutorial, NVIDIA NVFP4 introduction blog