https://duoan.github.io/tags/all-to-all/ Recent content in All-to-All on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sat, 10 May 2025 00:00:00 +0000 https://duoan.github.io/posts/moe-expert-parallelism-principles/ Sat, 10 May 2025 00:00:00 +0000 https://duoan.github.io/posts/moe-expert-parallelism-principles/ <h1 id="moe-parallelism-principles-gshard-expert-parallel-and-all-to-all">MoE Parallelism Principles: GShard, Expert Parallel, and All-to-All</h1> <p>Mixture-of-Experts is sparse compute wrapped in a communication problem. The model idea is simple: replace a dense FFN with many expert FFNs and route each token to a few of them. The distributed-training problem starts when those experts live on different GPUs.</p> <p>Shazeer et al. introduced the sparsely-gated MoE layer in <a href="https://arxiv.org/abs/1701.06538">Outrageously Large Neural Networks</a>. GShard made the pattern practical at scale with top-2 routing, capacity, load-balancing loss, and all-to-all dispatch in <a href="https://arxiv.org/abs/2006.16668">GShard</a>. Switch simplified the router to top-1 in <a href="https://arxiv.org/abs/2101.03961">Switch Transformers</a>. This post explains the shared systems model underneath them.</p>