https://duoan.github.io/tags/embeddings/ Recent content in Embeddings on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 19 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-model-parallel-internals/ Sat, 19 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/megatron-model-parallel-internals/ <h1 id="megatron-internals-ii-columnrow-parallel-linear-and-vocab-parallel-embedding">Megatron Internals II: Column/Row Parallel Linear and Vocab Parallel Embedding</h1> <p>Tensor parallelism is not &ldquo;split every tensor somehow.&rdquo; In Megatron, it is a small set of layer contracts: which dimension is local, which collective completes the dense math, and which gradient path communicates.</p> <p>The original <a href="https://arxiv.org/abs/1909.08053">Megatron-LM paper</a> is still the cleanest starting point: split transformer matrix multiplies so each GPU does useful dense GEMM, then communicate only where the algebra requires it. This post walks the implementation-level contracts behind <code>ColumnParallelLinear</code>, <code>RowParallelLinear</code>, <code>VocabParallelEmbedding</code>, and parallel cross entropy.</p>