https://duoan.github.io/tags/megascale/ Recent content in MegaScale on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 04 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/large-moe-from-sparsity-to-communication/ Sat, 04 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/large-moe-from-sparsity-to-communication/ <h1 id="large-moe-performance-the-three-walls-after-sparsity">Large MoE Performance: The Three Walls After Sparsity</h1> <p>Sparsity made MoE cheap on paper. At production scale it made training <em>harder</em> than dense: total parameters grow with <code>E</code>, per-token FLOPs grow with <code>k</code>, and the gap between those two numbers is exactly where systems break.</p> <p>The useful framing is not “optimize the MoE kernel.” It is the one NVIDIA’s Megatron-Core MoE report uses (<a href="https://arxiv.org/abs/2603.07685">arXiv:2603.07685</a>): <strong>Memory, Communication, and Compute Efficiency are three coupled walls.</strong> Push on one and pressure shows up in another. ByteDance’s MegaScale-MoE (<a href="https://arxiv.org/abs/2505.11432">arXiv:2505.11432</a>) proves the same thesis from the other direction — on 1,440 Hoppers, communication was ~44% of forward time before their redesign, and fixing parallelism + overlap delivered <strong>1.88×</strong> over Megatron-LM.</p>