https://duoan.github.io/tags/ulysses/ Recent content in Ulysses on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Mon, 19 May 2025 00:00:00 +0000 https://duoan.github.io/posts/sequence-parallelism-ulysses/ Mon, 19 May 2025 00:00:00 +0000 https://duoan.github.io/posts/sequence-parallelism-ulysses/ <h1 id="sequence-parallelism-ii-deepspeed-ulysses-and-all-to-all-attention">Sequence Parallelism II: DeepSpeed Ulysses and All-to-All Attention</h1> <p>Megatron SP is a careful memory optimization around an existing tensor-parallel block. DeepSpeed Ulysses starts from a different question. What if each device owns a sequence slice most of the time, but attention temporarily wants each device to own a head slice instead?</p> <p>The answer in DeepSpeed Ulysses is an All-to-All transpose (<a href="https://arxiv.org/abs/2309.14509">arXiv:2309.14509</a>). Before attention, every rank has all heads for a subset of tokens. After All-to-All, every rank has all tokens for a subset of heads. That one layout change lets local attention run per head while the rest of the layer can remain sequence-sharded.</p>