NVIDIA’s New MoE Kernels Are a Reminder That AI Cost Gets Won in the Epilogue

AI cost does not only get decided in model architecture diagrams. It gets decided in the epilogue of a grouped GEMM, in whether quantization causes another memory pass, and in whether the CPU is still being asked to schedule dynamic expert work while thousands of GPUs wait politely. Not the glamorous part of the stack. Also the part where the bill lives.

NVIDIA’s new post on CuTe DSL fused kernels for dense and mixture-of-experts training is one of those infrastructure notes that looks narrow until you remember how modern models are actually built. The company says its new fused MLP kernels deliver 1.3x to 2x kernel-level speedups over unfused paths, up to 8% end-to-end speedup in a DeepSeek-V3 pretraining setup, and up to 93% end-to-end speedup in a GPT-OSS pretraining setup. The 93% number is the headline. The more durable story is that MoE performance depends on eliminating memory traffic, synchronization, and launch overhead that the high-level model diagram politely ignores.

Mixture-of-experts models are attractive because they offer a bargain: increase total model capacity while activating only a subset of parameters per token. That bargain is real, but it is not free. Routing tokens to experts creates dynamic shapes. Expert loads are uneven. Grouped GEMMs need to know how many tokens each expert received. Activations, transposes, quantization, and communication can become the tax that eats the sparsity advantage. If the runtime is sloppy, an MoE model becomes a theoretical efficiency paper attached to a practical GPU bonfire.

NVIDIA identifies three bottlenecks directly: memory-bound activation functions, CPU overhead from runtime expert token counts, and memory-bound quantization passes. The new kernels, written with NVIDIA CuTe DSL, are available through cuDNN Frontend v1.23.0 and later, Transformer Engine v2.15 and later, and Megatron Core 26.04-alpha.rc2 and later. The initial sync-free MoE kernel family covers grouped GEMM plus quantize, grouped GEMM plus activation plus quantize or transpose, and grouped GEMM plus activation-gradient plus quantize or transpose.

Fusion is not just fewer kernels; it changes the execution model

The easy version of fusion is “combine operations so intermediate tensors do not round-trip through global memory.” That is already valuable. NVIDIA supports activation functions including SwiGLU, GeGLU, and sReLU, with options for clamping and scaling. GLU fusion is especially awkward because the input and gate chunks may be produced by different thread blocks. NVIDIA’s approach repacks weights so the same thread block can access both tiles and combine them inside the GEMM epilogue instead of writing out intermediate activations and reading them back later.

But the more important part is synchronization removal. MoE workloads are dynamic: tokens route to experts at runtime, and the system usually needs expert token counts before launching grouped GEMMs. If that information has to come back to the host CPU, you have created a dependency in the middle of the training iteration. That dependency is poison for CUDA Graph replay, where the whole point is to reduce launch overhead and make repeated execution predictable.

NVIDIA says these kernels track tokens per group inside GPU memory, avoiding the host-side synchronization normally used to retrieve expert shapes before launching dynamic grouped GEMMs. That enables full-iteration CUDA Graphs for sync-free MoE execution. This is the difference between an optimization that looks nice in a microbenchmark and one that changes how the training loop behaves at scale. The CPU stops being part of the critical path for dynamic expert scheduling. That matters when the “small” overhead repeats across every layer, every microbatch, every step, for weeks.

The quantization work follows the same pattern. For MXFP8, NVIDIA folds quantization and transpose into the GEMM kernel, avoiding an extra BF16 activation read and write. For NVFP4, the kernel produces BF16 output and per-tensor amax for forward propagation, and computes amax for the transposed Hadamard-rotated output during backprop. Translation: the model gets the low-precision path without sprinkling extra memory-bound cleanup kernels all over the iteration.

The 93% number needs context; the 8% number may be the planning number

NVIDIA reports forward-pass speedups up to 1.3x and backward-pass speedups up to 2.1x in unit microbenchmarks. At the full-training level, the company reports up to 8% improvement on DeepSeek-V3 pretraining and up to 93% improvement on GPT-OSS pretraining. Both are useful, but they are not the same kind of claim.

A 93% end-to-end gain usually means the baseline was bottlenecked hard by synchronization, launch overhead, memory traffic, or an immature path. That does not make the result fake; it makes it situational. If your stack looks like that baseline, this kind of optimization can be enormous. If your stack is already heavily tuned, the DeepSeek-V3 8% figure may be closer to what you should plan around. And 8% is still a lot. On a long pretraining run, 8% is not a rounding error. It is days or weeks of cluster time, megawatt-hours of power, and enough money that someone in finance will suddenly become interested in epilogues.

The practical move is to reproduce, not admire. NVIDIA points to benchmark code in cuDNN Frontend comparing fused grouped-GEMM plus activation kernels against the equivalent unfused Transformer Engine path using CUDA Graph replay. The default benchmark shape is 8 experts, 4096 tokens per expert, K=8192, N=4096, BF16, using torch 2.12.0a0+nv26.05, transformer-engine 2.15.0, nvidia-cudnn-frontend 1.23.0, and nvidia-cutlass-dsl 4.4.1. That is specific enough for serious teams to test on their own hardware instead of screenshotting a chart into a planning doc.

If you train or fine-tune MoE models on NVIDIA GPUs, the checklist is straightforward. Confirm whether your framework path actually picks up these fused kernels through cuDNN Frontend, Transformer Engine, or Megatron Core. Run the provided microbenchmarks on the target GPUs and precisions you use. Profile with Nsight Systems before and after. Look for exposed activation kernels, extra quantization passes, host synchronization gaps, launch overhead, and Tensor Core idle regions between memory-bound operations. If the profiler shows the GPU waiting between grouped GEMM, activation, transpose, and quantization, the architecture diagram has been lying to you politely.

This also matters beyond pretraining labs. The coding-agent and open-weight model ecosystem is increasingly MoE-shaped because MoE is one route to high capacity under practical compute budgets. Better MoE training and runtime efficiency can lower the cost of producing and adapting competitive models. That affects whether organizations can self-host capable coding models, whether open-weight models can keep pressure on API vendors, and whether “best AI coding agent” comparisons are about model quality alone or model-plus-runtime economics. Spoiler: it is the latter.

The lock-in boundary is worth naming. CuTe DSL, cuDNN Frontend, Transformer Engine, Megatron Core, MXFP8, NVFP4, CUDA Graph assumptions — this is NVIDIA’s stack, deeply. Hardware-specific optimization is not a moral failure; it is how performance happens. But practitioners should be honest that “open model” and “portable economics” are different claims. A checkpoint may be downloadable while the fastest training path depends on vendor-specific kernels and precision formats. That is not necessarily a reason to avoid the stack. It is a reason to price the dependency correctly.

The senior-engineer takeaway: stop evaluating MoE systems only at the Python layer. Ask what happens between grouped GEMM, activation, quantization, transpose, and expert-parallel communication. Ask whether the full iteration can be captured in CUDA Graphs. Ask whether the CPU is still scheduling dynamic expert work. Those questions sound too low-level until the training bill arrives. Then they sound like product strategy.

Sources: NVIDIA Developer Blog, cuDNN Frontend CuTe DSL fusion benchmarks, NVIDIA Transformer Engine, NVIDIA Megatron-LM