Communication

Large MoE Performance: The Three Walls After Sparsity Sparsity made MoE cheap on paper. At production scale it made training harder than dense: total parameters grow with E, per-token FLOPs grow with k, and the gap between those two numbers is exactly where systems break. The useful framing is not “optimize the MoE kernel.” It is the one NVIDIA’s Megatron-Core MoE report uses (arXiv:2603.07685): Memory, Communication, and Compute Efficiency are three coupled walls. Push on one and pressure shows up in another. ByteDance’s MegaScale-MoE (arXiv:2505.11432) proves the same thesis from the other direction — on 1,440 Hoppers, communication was ~44% of forward time before their redesign, and fixing parallelism + overlap delivered 1.88× over Megatron-LM. ...

Hiding Tensor-Parallel Collectives: AG/RS Overlap in Megatron Once Megatron SP is enabled, tensor-parallel communication often appears as AllGather and ReduceScatter rather than a single AllReduce. That is a memory win because activations can stay sequence-sharded between tensor-parallel regions. It also creates a scheduling question. Can the collectives be hidden under GEMM work? That question sits directly on top of Megatron’s tensor/sequence-parallel training path (arXiv:2104.04473, arXiv:2205.05198). This post is about that scheduling question. It complements Megatron tensor parallelism and Megatron SP. The goal is not to memorize every flag in Megatron or Transformer Engine. The goal is to recognize where the dependency graph allows overlap and where it does not. ...