https://duoan.github.io/tags/pipeline-parallelism/ Recent content in Pipeline Parallelism on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sun, 16 Mar 2025 00:00:00 +0000 https://duoan.github.io/posts/pipeline-parallelism-gpipe/ Sun, 16 Mar 2025 00:00:00 +0000 https://duoan.github.io/posts/pipeline-parallelism-gpipe/ <h1 id="pipeline-parallelism-from-first-principles-why-gpipe-split-the-batch">Pipeline Parallelism from First Principles: Why GPipe Split the Batch</h1> <p>Pipeline parallelism starts with a useful accident: a Transformer is already a chain. If one GPU cannot hold the whole stack, put consecutive blocks on consecutive GPUs and send activations across stage boundaries.</p> <p>That solves capacity. It does not solve throughput. A naive layer split turns an expensive cluster into a queue where most devices wait. GPipe&rsquo;s contribution was to make the queue busy by slicing the mini-batch into micro-batches and recomputing activations instead of storing everything (<a href="https://arxiv.org/abs/1811.06965">arXiv:1811.06965</a>).</p>