https://duoan.github.io/tags/data-parallelism/ Recent content in Data Parallelism on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Sun, 06 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/data-parallelism-ddp-ring-allreduce/ Sun, 06 Apr 2025 00:00:00 +0000 https://duoan.github.io/posts/data-parallelism-ddp-ring-allreduce/ <h1 id="data-parallelism-from-parameter-server-to-ring-all-reduce">Data Parallelism: From Parameter Server to Ring All-Reduce</h1> <p>Data parallelism is the default scaling move because it preserves the model program. Every rank owns the same model, sees different examples, computes gradients, and applies the same update.</p> <p>The algorithm is simple. The system is not. The whole post is about removing one bottleneck: do not push all gradient traffic through one server when every GPU could be moving bytes at the same time.</p>