https://duoan.github.io/tags/context-parallelism/ Recent content in Context Parallelism on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Mon, 02 Jun 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-context-parallel/ Mon, 02 Jun 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-context-parallel/ <h1 id="sequence-parallelism-iv-megatron-context-parallel-and-load-balanced-rings">Sequence Parallelism IV: Megatron Context Parallel and Load-Balanced Rings</h1> <p>Megatron Context Parallel, usually shortened to CP, brings long-context attention into Megatron&rsquo;s hybrid-parallel world. It keeps the familiar tensor, pipeline, and data-parallel axes, then adds a context axis that shards the sequence. Inside that context group, attention behaves like a ring, borrowing the same blockwise K/V circulation idea as Ring Attention (<a href="https://arxiv.org/abs/2310.01889">arXiv:2310.01889</a>) while staying compatible with Megatron-style tensor and pipeline parallelism (<a href="https://arxiv.org/abs/2104.04473">arXiv:2104.04473</a>).</p>