Posts

Data Parallelism: From Parameter Server to Ring All-Reduce Data parallelism is the default scaling move because it preserves the model program. Every rank owns the same model, sees different examples, computes gradients, and applies the same update. The algorithm is simple. The system is not. The whole post is about removing one bottleneck: do not push all gradient traffic through one server when every GPU could be moving bytes at the same time. ...

Pipeline Parallelism from First Principles: Why GPipe Split the Batch Pipeline parallelism starts with a useful accident: a Transformer is already a chain. If one GPU cannot hold the whole stack, put consecutive blocks on consecutive GPUs and send activations across stage boundaries. That solves capacity. It does not solve throughput. A naive layer split turns an expensive cluster into a queue where most devices wait. GPipe’s contribution was to make the queue busy by slicing the mini-batch into micro-batches and recomputing activations instead of storing everything (arXiv:1811.06965). ...

LLM Training Series: A Systems Map for Distributed Transformers LLM training stops being “run the model on more GPUs” the moment one replica no longer fits, one link becomes the step time, or one schedule leaves half the cluster idle. From that point on, the training run is a placement problem. The stable question is simple: which bytes are replicated, which bytes are sharded, and which link moves them on the critical path? Data parallelism, tensor parallelism, pipeline parallelism, ZeRO, sequence/context parallelism, and expert parallelism are different answers to that question. The public systems that matter - GPipe, Megatron-LM, DeepSpeed ZeRO, PyTorch DDP, GShard, Switch Transformer, and modern Megatron-Core - all expose the same constraints in different shapes (GPipe, Megatron-LM, ZeRO, PyTorch Distributed, GShard, Switch Transformer). ...

From Scaling Laws to Cluster Size: Capacity Planning That Survives Contact With GPUs Scaling laws tell you how loss should move with tokens and parameters. Clusters tell you what you can actually buy. Capacity planning is the bridge — and most teams skip it until the first failed launch burns a week of calendar time and a pile of GPU-hours. This post is a reproducible planning pipeline: tokens → steps → FLOPs → GPU-hours → forced parallelism. The arithmetic lives in playground/capacity_plan.py; the figures below are generated from its JSON output. Not exact. Accurate enough to catch 10× fantasy plans. ...