Roofline: The First Step of Any Performance Optimization
Roofline: The First Step of Any Performance Optimization When MFU sits at 20%, most people open a profiler and hunt for a slow kernel. That often starts at the wrong layer. The first question is not which kernel is hot — it is which ceiling you are hitting: compute or memory bandwidth. TL;DR Every GPU has two hard ceilings: peak FLOP/s and peak bandwidth. Arithmetic intensity I = FLOPs / Bytes decides which one binds first. MFU answers “are we compute-bound?” for training. MBU answers “are we bandwidth-bound?” for decode. Both are Roofline ratios, not vibes. Shape matters more than op name: the same matmul can be compute-bound at M=N=K=8192 and memory-bound at M=1. That is why training and decode feel like different worlds. Count MFU/MBU by instrumentation (FlopCounterMode + bytes), not PaLM 6PT — that formula is an LLM shortcut. ResNet / ViT work the same way as any other nn.Module. Model-level Roofline is useful when traffic is homogeneous (decode, dense training GEMMs). It is misleading when time is dominated by a mix of memory-bound and compute-bound ops — then go per-op, then profiler. Reproducible Modal measurements (ops, LLM decode/train sweeps, ResNet/ViT MFU·MBU) live in this page bundle; code in playground/roofline_modal.py. The Most Expensive Mistake The costly failure mode in performance work is not missing the optimal kernel. It is optimizing in the wrong direction. ...
