Distributed Training

ZeRO: Partitioning Optimizer State, Gradients, and Parameters Plain DDP is clean but wasteful. Every data-parallel rank stores the same parameters, gradients, fp32 master weights, and optimizer moments. At small scale that redundancy is convenient. At LLM scale it is the memory wall. ZeRO, the Zero Redundancy Optimizer, keeps data-parallel semantics and removes the redundant storage one category at a time. ZeRO-1 shards optimizer state. ZeRO-2 shards optimizer state and gradients. ZeRO-3 shards optimizer state, gradients, and parameters (arXiv:1910.02054). ...

Megatron Internals I: Building the DP / TP / PP Process Groups Megatron-LM’s first trick is not tensor-parallel matmul. It is rank bookkeeping. Before the model runs, every process must know the small set of peers it will communicate with for tensor parallelism, pipeline parallelism, data parallelism, embeddings, and optimizer state. The original Megatron-LM paper introduced intra-layer tensor parallelism for transformer training. The later Megatron-LM scaling paper put tensor parallelism, pipeline parallelism, and data parallelism into one training system. This post is the control plane underneath that system: how a flat list of ranks becomes a DP / PP / TP mesh. ...

Data Parallelism: From Parameter Server to Ring All-Reduce Data parallelism is the default scaling move because it preserves the model program. Every rank owns the same model, sees different examples, computes gradients, and applies the same update. The algorithm is simple. The system is not. The whole post is about removing one bottleneck: do not push all gradient traffic through one server when every GPU could be moving bytes at the same time. ...

Pipeline Parallelism from First Principles: Why GPipe Split the Batch Pipeline parallelism starts with a useful accident: a Transformer is already a chain. If one GPU cannot hold the whole stack, put consecutive blocks on consecutive GPUs and send activations across stage boundaries. That solves capacity. It does not solve throughput. A naive layer split turns an expensive cluster into a queue where most devices wait. GPipe’s contribution was to make the queue busy by slicing the mini-batch into micro-batches and recomputing activations instead of storing everything (arXiv:1811.06965). ...

LLM Training Series: A Systems Map for Distributed Transformers LLM training stops being “run the model on more GPUs” the moment one replica no longer fits, one link becomes the step time, or one schedule leaves half the cluster idle. From that point on, the training run is a placement problem. The stable question is simple: which bytes are replicated, which bytes are sharded, and which link moves them on the critical path? Data parallelism, tensor parallelism, pipeline parallelism, ZeRO, sequence/context parallelism, and expert parallelism are different answers to that question. The public systems that matter - GPipe, Megatron-LM, DeepSpeed ZeRO, PyTorch DDP, GShard, Switch Transformer, and modern Megatron-Core - all expose the same constraints in different shapes (GPipe, Megatron-LM, ZeRO, PyTorch Distributed, GShard, Switch Transformer). ...