https://duoan.github.io/categories/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/cuda/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/diffusion/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/categories/engineering/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/flashattention/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/posts/foundation-model-kernel-optimization-2026/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/kernel/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/llm/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/moe/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/categories/performance/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/performance/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/posts/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/systems/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/vlm/ 2026-07-16T00:00:00+00:00 https://duoan.github.io/tags/distributed-training/ 2026-07-13T00:00:00+00:00 https://duoan.github.io/tags/paper-reading/ 2026-07-13T00:00:00+00:00 https://duoan.github.io/posts/argus-tracing-at-10000-gpu-scale/ 2026-07-13T00:00:00+00:00 https://duoan.github.io/tags/profiling/ 2026-07-13T00:00:00+00:00 https://duoan.github.io/tags/gpu/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/memory/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/optimization/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/posts/gpu-optimization-playbook/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/mfu/ 2026-07-11T00:00:00+00:00 https://duoan.github.io/tags/modal/ 2026-07-11T00:00:00+00:00 https://duoan.github.io/tags/roofline/ 2026-07-11T00:00:00+00:00 https://duoan.github.io/posts/roofline-first-step-of-performance-optimization/ 2026-07-11T00:00:00+00:00 https://duoan.github.io/tags/communication/ 2026-07-04T00:00:00+00:00 https://duoan.github.io/tags/grouped-gemm/ 2026-07-04T00:00:00+00:00 https://duoan.github.io/posts/large-moe-from-sparsity-to-communication/ 2026-07-04T00:00:00+00:00 https://duoan.github.io/tags/megascale/ 2026-07-04T00:00:00+00:00 https://duoan.github.io/tags/megatron/ 2026-07-04T00:00:00+00:00 https://duoan.github.io/tags/dynamo/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/tags/inductor/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/tags/pytorch/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/tags/torch.compile/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/posts/torch-compile-from-bytecode-to-triton/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/tags/triton/ 2026-05-31T00:00:00+00:00 https://duoan.github.io/tags/bf16/ 2026-05-24T00:00:00+00:00 https://duoan.github.io/tags/kernel-fusion/ 2026-05-24T00:00:00+00:00 https://duoan.github.io/posts/optimizing-vlm-training-on-one-gpu/ 2026-05-24T00:00:00+00:00 https://duoan.github.io/tags/training/ 2026-05-24T00:00:00+00:00 https://duoan.github.io/tags/data-pipeline/ 2026-03-12T00:00:00+00:00 https://duoan.github.io/tags/ddp/ 2026-03-12T00:00:00+00:00 https://duoan.github.io/tags/transformer/ 2026-03-12T00:00:00+00:00 https://duoan.github.io/posts/why-variable-sequence-length-breaks-ddp-throughput/ 2026-03-12T00:00:00+00:00 https://duoan.github.io/posts/learning-ddp-performance-tuning-on-one-gpu/ 2026-02-18T00:00:00+00:00 https://duoan.github.io/tags/nsight/ 2026-01-16T00:00:00+00:00 https://duoan.github.io/posts/profiling-pytorch-training-end-to-end/ 2026-01-16T00:00:00+00:00 https://duoan.github.io/posts/scalable-multimodal-datapipeline/ 2025-12-22T00:00:00+00:00 https://duoan.github.io/tags/multimodal/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/categories/research/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/scalability/ 2025-12-22T00:00:00+00:00 https://duoan.github.io/tags/streaming/ 2025-12-22T00:00:00+00:00 https://duoan.github.io/tags/cot/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/curriculum/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/distillation/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/qwen/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/siglip/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/siq-vl/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/posts/siq-vl-curriculum-under-compute-constraints/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/deepspeed/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/tags/distributed-systems/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/series/llm-training/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/series/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/posts/zero3-intra-layer-partitioning/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/tags/zero/ 2025-06-16T00:00:00+00:00 https://duoan.github.io/posts/megatron-tp-comm-overlap/ 2025-06-09T00:00:00+00:00 https://duoan.github.io/tags/parallelism/ 2025-06-09T00:00:00+00:00 https://duoan.github.io/tags/context-parallelism/ 2025-06-02T00:00:00+00:00 https://duoan.github.io/posts/megatron-context-parallel/ 2025-06-02T00:00:00+00:00 https://duoan.github.io/tags/attention/ 2025-05-26T00:00:00+00:00 https://duoan.github.io/tags/long-context/ 2025-05-26T00:00:00+00:00 https://duoan.github.io/posts/ring-attention/ 2025-05-26T00:00:00+00:00 https://duoan.github.io/posts/sequence-parallelism-ulysses/ 2025-05-19T00:00:00+00:00 https://duoan.github.io/tags/ulysses/ 2025-05-19T00:00:00+00:00 https://duoan.github.io/tags/megatron-lm/ 2025-05-18T00:00:00+00:00 https://duoan.github.io/tags/nccl/ 2025-05-18T00:00:00+00:00 https://duoan.github.io/tags/tensor-parallelism/ 2025-05-18T00:00:00+00:00 https://duoan.github.io/posts/tensor-parallelism-megatron/ 2025-05-18T00:00:00+00:00 https://duoan.github.io/tags/expert-parallel/ 2025-05-17T00:00:00+00:00 https://duoan.github.io/posts/moe-deepspeed-megatron-internals/ 2025-05-17T00:00:00+00:00 https://duoan.github.io/tags/sequence-parallelism/ 2025-05-12T00:00:00+00:00 https://duoan.github.io/posts/sequence-parallelism-megatron-sp/ 2025-05-12T00:00:00+00:00 https://duoan.github.io/tags/all-to-all/ 2025-05-10T00:00:00+00:00 https://duoan.github.io/tags/gshard/ 2025-05-10T00:00:00+00:00 https://duoan.github.io/posts/moe-expert-parallelism-principles/ 2025-05-10T00:00:00+00:00 https://duoan.github.io/tags/optimizer/ 2025-04-27T00:00:00+00:00 https://duoan.github.io/posts/zero-redundancy-optimizer/ 2025-04-27T00:00:00+00:00 https://duoan.github.io/tags/amp/ 2025-04-26T00:00:00+00:00 https://duoan.github.io/tags/loss-scaling/ 2025-04-26T00:00:00+00:00 https://duoan.github.io/posts/megatron-mixed-precision-training/ 2025-04-26T00:00:00+00:00 https://duoan.github.io/tags/mixed-precision/ 2025-04-26T00:00:00+00:00 https://duoan.github.io/tags/embeddings/ 2025-04-19T00:00:00+00:00 https://duoan.github.io/posts/megatron-model-parallel-internals/ 2025-04-19T00:00:00+00:00 https://duoan.github.io/tags/model-parallel/ 2025-04-19T00:00:00+00:00 https://duoan.github.io/tags/tensor-parallel/ 2025-04-19T00:00:00+00:00 https://duoan.github.io/tags/data-parallel/ 2025-04-12T00:00:00+00:00 https://duoan.github.io/posts/megatron-distributed-init/ 2025-04-12T00:00:00+00:00 https://duoan.github.io/tags/pipeline-parallel/ 2025-04-12T00:00:00+00:00 https://duoan.github.io/tags/all-reduce/ 2025-04-06T00:00:00+00:00 https://duoan.github.io/tags/data-parallelism/ 2025-04-06T00:00:00+00:00 https://duoan.github.io/posts/data-parallelism-ddp-ring-allreduce/ 2025-04-06T00:00:00+00:00 https://duoan.github.io/tags/gpipe/ 2025-03-16T00:00:00+00:00 https://duoan.github.io/tags/pipeline-parallelism/ 2025-03-16T00:00:00+00:00 https://duoan.github.io/posts/pipeline-parallelism-gpipe/ 2025-03-16T00:00:00+00:00 https://duoan.github.io/posts/llm-training-series/ 2025-03-02T00:00:00+00:00 https://duoan.github.io/tags/capacity-planning/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/posts/large-model-capacity-plan/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/tags/scaling-laws/ 2026-07-12T00:00:00+00:00 https://duoan.github.io/archives/ https://duoan.github.io/search/