GPU

The GPU Optimization Playbook: Architecture, Memory, and Balance Most “GPU optimization” advice is a bag of tricks: coalesce here, unroll there, add __restrict__ and pray. Tricks are the output of optimization, not the method. The method is smaller and more durable: understand the machine, find the resource that is actually saturated, and rebalance work toward the resources that are idle. Almost every GPU kernel is limited by data movement, not arithmetic. Once you internalize that, the whole catalog of techniques collapses into three questions. ...

Roofline: The First Step of Any Performance Optimization When MFU sits at 20%, most people open a profiler and hunt for a slow kernel. That often starts at the wrong layer. The first question is not which kernel is hot — it is which ceiling you are hitting: compute or memory bandwidth. TL;DR Every GPU has two hard ceilings: peak FLOP/s and peak bandwidth. Arithmetic intensity I = FLOPs / Bytes decides which one binds first. MFU answers “are we compute-bound?” for training. MBU answers “are we bandwidth-bound?” for decode. Both are Roofline ratios, not vibes. Shape matters more than op name: the same matmul can be compute-bound at M=N=K=8192 and memory-bound at M=1. That is why training and decode feel like different worlds. Count MFU/MBU by instrumentation (FlopCounterMode + bytes), not PaLM 6PT — that formula is an LLM shortcut. ResNet / ViT work the same way as any other nn.Module. Model-level Roofline is useful when traffic is homogeneous (decode, dense training GEMMs). It is misleading when time is dominated by a mix of memory-bound and compute-bound ops — then go per-op, then profiler. Reproducible Modal measurements (ops, LLM decode/train sweeps, ResNet/ViT MFU·MBU) live in this page bundle; code in playground/roofline_modal.py. The Most Expensive Mistake The costly failure mode in performance work is not missing the optimal kernel. It is optimizing in the wrong direction. ...

Optimizing VLM Training on One GPU: A Five-Layer Recipe How I got SiQ-VL from 14,713 to 100,923 real tokens per second on a single Blackwell GPU, and the four places that surprised me along the way. TL;DR I trained a small vision-language model (SigLIP-2 vision tower + Qwen2.5 LLM, projector-aligned) on one NVIDIA RTX PRO 6000 Blackwell, ran a 48-configuration sweep across two model sizes and both training stages, and ended up with a recipe that compounds five optimization layers. ...