https://duoan.github.io/tags/long-context/ Recent content in Long Context on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Mon, 26 May 2025 00:00:00 +0000 https://duoan.github.io/posts/ring-attention/ Mon, 26 May 2025 00:00:00 +0000 https://duoan.github.io/posts/ring-attention/ <h1 id="sequence-parallelism-iii-ring-attention-for-context-that-does-not-fit">Sequence Parallelism III: Ring Attention for Context That Does Not Fit</h1> <p>Megatron SP reduces replicated activation memory around tensor-parallel blocks. DeepSpeed Ulysses uses All-to-All to turn sequence shards into head shards for attention. Ring Attention changes the unit of work again. It asks each rank to keep a block of queries fixed, then circulate key/value blocks around a ring until every query block has seen every key/value block it needs. That is the core idea in <em>Ring Attention with Blockwise Transformers for Near-Infinite Context</em> (<a href="https://arxiv.org/abs/2310.01889">arXiv:2310.01889</a>).</p>