https://duoan.github.io/tags/sequence-parallelism/ Recent content in Sequence Parallelism on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Mon, 12 May 2025 00:00:00 +0000 https://duoan.github.io/posts/sequence-parallelism-megatron-sp/ Mon, 12 May 2025 00:00:00 +0000 https://duoan.github.io/posts/sequence-parallelism-megatron-sp/ <h1 id="sequence-parallelism-i-megatron-sp-cuts-activation-memory-along-the-sequence">Sequence Parallelism I: Megatron SP Cuts Activation Memory Along the Sequence</h1> <p>Tensor parallelism is usually introduced as a way to make matrix multiplications fit. That is true, but it hides a second problem. After the weights are split, many activations are still replicated on every tensor-parallel rank. For short contexts this is tolerable. For long contexts it becomes one of the reasons training throughput collapses into activation checkpointing.</p> <p>Megatron sequence parallelism, usually shortened to Megatron SP, is a targeted fix from <em>Reducing Activation Recomputation in Large Transformer Models</em> (<a href="https://arxiv.org/abs/2205.05198">arXiv:2205.05198</a>). It does not replace tensor parallelism. It keeps Megatron&rsquo;s column-parallel and row-parallel linear layers, then shards the sequence-local regions that tensor parallelism had left replicated. The trick is small enough to miss and important enough to change the memory budget of a whole Transformer block.</p>