https://duoan.github.io/tags/tensor-parallelism/ Recent content in Tensor Parallelism on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sun, 18 May 2025 00:00:00 +0000 https://duoan.github.io/posts/tensor-parallelism-megatron/ Sun, 18 May 2025 00:00:00 +0000 https://duoan.github.io/posts/tensor-parallelism-megatron/ <h1 id="tensor-parallelism-in-megatron-lm-splitting-layers-not-stacks">Tensor Parallelism in Megatron-LM: Splitting Layers, Not Stacks</h1> <p>Pipeline parallelism splits a model by depth. Tensor parallelism splits the math inside one layer.</p> <p>Megatron-LM made tensor parallelism practical for Transformers by choosing split points that preserve local GEMMs and put collectives only where the algebra requires them. The original Megatron paper introduced the core intra-layer pattern (<a href="https://arxiv.org/abs/1909.08053">arXiv:1909.08053</a>). The later Megatron-LM systems paper showed how that TP axis composes with data and pipeline parallelism at cluster scale (<a href="https://arxiv.org/abs/2104.04473">arXiv:2104.04473</a>).</p>