https://duoan.github.io/tags/mfu/ Recent content in MFU on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 11 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/roofline-first-step-of-performance-optimization/ Sat, 11 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/roofline-first-step-of-performance-optimization/ <h1 id="roofline-the-first-step-of-any-performance-optimization">Roofline: The First Step of Any Performance Optimization</h1> <p>When MFU sits at 20%, most people open a profiler and hunt for a slow kernel. That often starts at the wrong layer. The first question is not <em>which kernel is hot</em> — it is <em>which ceiling you are hitting</em>: compute or memory bandwidth.</p> <h2 id="tldr">TL;DR</h2> <ul> <li>Every GPU has two hard ceilings: peak FLOP/s and peak bandwidth. Arithmetic intensity <code>I = FLOPs / Bytes</code> decides which one binds first.</li> <li>MFU answers “are we compute-bound?” for training. MBU answers “are we bandwidth-bound?” for decode. Both are Roofline ratios, not vibes.</li> <li>Shape matters more than op name: the same <code>matmul</code> can be compute-bound at <code>M=N=K=8192</code> and memory-bound at <code>M=1</code>. That is why training and decode feel like different worlds.</li> <li>Count MFU/MBU by instrumentation (<code>FlopCounterMode</code> + bytes), not PaLM <code>6PT</code> — that formula is an LLM shortcut. ResNet / ViT work the same way as any other <code>nn.Module</code>.</li> <li>Model-level Roofline is useful when traffic is homogeneous (decode, dense training GEMMs). It is misleading when time is dominated by a mix of memory-bound and compute-bound ops — then go per-op, then profiler.</li> <li>Reproducible Modal measurements (ops, LLM decode/train sweeps, ResNet/ViT MFU·MBU) live in this page bundle; code in <a href="https://github.com/duoan/duoan.github.io/blob/main/playground/roofline_modal.py"><code>playground/roofline_modal.py</code></a>.</li> </ul> <h2 id="the-most-expensive-mistake">The Most Expensive Mistake</h2> <p>The costly failure mode in performance work is not missing the optimal kernel. It is optimizing in the wrong direction.</p>