CUDA

Foundation Model Kernel Optimization in 2026: A Field Guide Across Dense, MoE, Multimodal, and Diffusion Most “kernel optimization” conversations still start with FLOPs. In 2026 that is usually the wrong first question. Foundation-model runtime is dominated by HBM traffic, KV cache, temporary tensors, collectives, dynamic permutation, and launch overhead. FlashAttention, fused linear–cross-entropy, paged KV, MoE grouped GEMM, and DeepEP-style dispatch all share one essence: do not materialize intermediates, or make each byte travel once. ...

The GPU Optimization Playbook: Architecture, Memory, and Balance Most “GPU optimization” advice is a bag of tricks: coalesce here, unroll there, add __restrict__ and pray. Tricks are the output of optimization, not the method. The method is smaller and more durable: understand the machine, find the resource that is actually saturated, and rebalance work toward the resources that are idle. Almost every GPU kernel is limited by data movement, not arithmetic. Once you internalize that, the whole catalog of techniques collapses into three questions. ...

Roofline: The First Step of Any Performance Optimization When MFU sits at 20%, most people open a profiler and hunt for a slow kernel. That often starts at the wrong layer. The first question is not which kernel is hot — it is which ceiling you are hitting: compute or memory bandwidth. TL;DR Every GPU has two hard ceilings: peak FLOP/s and peak bandwidth. Arithmetic intensity I = FLOPs / Bytes decides which one binds first. MFU answers “are we compute-bound?” for training. MBU answers “are we bandwidth-bound?” for decode. Both are Roofline ratios, not vibes. Shape matters more than op name: the same matmul can be compute-bound at M=N=K=8192 and memory-bound at M=1. That is why training and decode feel like different worlds. Count MFU/MBU by instrumentation (FlopCounterMode + bytes), not PaLM 6PT — that formula is an LLM shortcut. ResNet / ViT work the same way as any other nn.Module. Model-level Roofline is useful when traffic is homogeneous (decode, dense training GEMMs). It is misleading when time is dominated by a mix of memory-bound and compute-bound ops — then go per-op, then profiler. Reproducible Modal measurements (ops, LLM decode/train sweeps, ResNet/ViT MFU·MBU) live in this page bundle; code in playground/roofline_modal.py. The Most Expensive Mistake The costly failure mode in performance work is not missing the optimal kernel. It is optimizing in the wrong direction. ...

Profiling a PyTorch Training Job End to End How to decide whether a training job is blocked on data, PyTorch overhead, or a hot CUDA kernel, using torch.profiler, Nsight Systems, and Nsight Compute in the right order. TL;DR When a PyTorch training job feels slow, the most expensive mistake is starting at the wrong layer. In this case study, I built a small synthetic training lab and used it to force three common bottlenecks: ...