https://duoan.github.io/tags/pipeline-parallel/ Recent content in Pipeline Parallel on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 12 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-distributed-init/ Sat, 12 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-distributed-init/ <h1 id="megatron-internals-i-building-the-dp--tp--pp-process-groups">Megatron Internals I: Building the DP / TP / PP Process Groups</h1> <p>Megatron-LM&rsquo;s first trick is not tensor-parallel matmul. It is rank bookkeeping. Before the model runs, every process must know the small set of peers it will communicate with for tensor parallelism, pipeline parallelism, data parallelism, embeddings, and optimizer state.</p> <p>The original <a href="https://arxiv.org/abs/1909.08053">Megatron-LM paper</a> introduced intra-layer tensor parallelism for transformer training. The later <a href="https://arxiv.org/abs/2104.04473">Megatron-LM scaling paper</a> put tensor parallelism, pipeline parallelism, and data parallelism into one training system. This post is the control plane underneath that system: how a flat list of ranks becomes a DP / PP / TP mesh.</p>