https://duoan.github.io/tags/tensor-parallel/ Recent content in Tensor Parallel on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 19 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-model-parallel-internals/ Sat, 19 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-model-parallel-internals/ <h1 id="megatron-internals-ii-columnrow-parallel-linear-and-vocab-parallel-embedding">Megatron Internals II: Column/Row Parallel Linear and Vocab Parallel Embedding</h1> <p>Tensor parallelism is not &ldquo;split every tensor somehow.&rdquo; In Megatron, it is a small set of layer contracts: which dimension is local, which collective completes the dense math, and which gradient path communicates.</p> <p>The original <a href="https://arxiv.org/abs/1909.08053">Megatron-LM paper</a> is still the cleanest starting point: split transformer matrix multiplies so each GPU does useful dense GEMM, then communicate only where the algebra requires it. This post walks the implementation-level contracts behind <code>ColumnParallelLinear</code>, <code>RowParallelLinear</code>, <code>VocabParallelEmbedding</code>, and parallel cross entropy.</p> https://duoan.github.io/posts/megatron-distributed-init/ Sat, 12 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-distributed-init/ <h1 id="megatron-internals-i-building-the-dp--tp--pp-process-groups">Megatron Internals I: Building the DP / TP / PP Process Groups</h1> <p>Megatron-LM&rsquo;s first trick is not tensor-parallel matmul. It is rank bookkeeping. Before the model runs, every process must know the small set of peers it will communicate with for tensor parallelism, pipeline parallelism, data parallelism, embeddings, and optimizer state.</p> <p>The original <a href="https://arxiv.org/abs/1909.08053">Megatron-LM paper</a> introduced intra-layer tensor parallelism for transformer training. The later <a href="https://arxiv.org/abs/2104.04473">Megatron-LM scaling paper</a> put tensor parallelism, pipeline parallelism, and data parallelism into one training system. This post is the control plane underneath that system: how a flat list of ranks becomes a DP / PP / TP mesh.</p>