Archive

2026  10

July  5

Foundation Model Kernel Optimization in 2026: A Field Guide Across Dense, MoE, Multimodal, and Diffusion

July 16, 2026 · 36 min · Duo An

Paper Reading: ARGUS — Always-On Tracing at 10,000+ GPU Scale

July 13, 2026 · 15 min · Duo An

The GPU Optimization Playbook: Architecture, Memory, and Balance

July 12, 2026 · 10 min · Duo An

Roofline: The First Step of Any Performance Optimization

July 11, 2026 · 12 min · Duo An

Large MoE Performance: The Three Walls After Sparsity

July 4, 2026 · 12 min · Duo An

May  2

torch.compile: The Mental Model That Actually Matters

May 31, 2026 · 9 min · Duo An

Optimizing VLM Training on One GPU: A Five-Layer Recipe

May 24, 2026 · 14 min · Duo An

March  1

Why Variable Sequence Length Breaks DDP Throughput

March 12, 2026 · 8 min · Duo An

February  1

Learning PyTorch DDP Performance Tuning on a One-GPU Machine

February 18, 2026 · 15 min · Duo An

January  1

Profiling a PyTorch Training Job End to End

January 16, 2026 · 12 min · Duo An

2025  19

December  2

Building a Truly Scalable Multimodal Data Pipeline: A Streaming-First View

December 22, 2025 · 4 min · Duo An

SiQ-VL: A Curriculum for Small VLMs When Compute Is the Hard Constraint

December 15, 2025 · 6 min · Duo An

June  3

The ZeRO-3 Diagram Most People Remember Is Wrong

June 16, 2025 · 8 min · Duo An

Hiding Tensor-Parallel Collectives: AG/RS Overlap in Megatron

June 9, 2025 · 8 min · Duo An

Sequence Parallelism IV: Megatron Context Parallel and Load-Balanced Rings

June 2, 2025 · 8 min · Duo An

May  6

Sequence Parallelism III: Ring Attention for Context That Does Not Fit

May 26, 2025 · 8 min · Duo An

Sequence Parallelism II: DeepSpeed Ulysses and All-to-All Attention

May 19, 2025 · 8 min · Duo An

Tensor Parallelism in Megatron-LM: Splitting Layers, Not Stacks

May 18, 2025 · 8 min · Duo An

MoE Internals: DeepSpeed-Megatron Expert Parallel Implementation

May 17, 2025 · 8 min · Duo An

Sequence Parallelism I: Megatron SP Cuts Activation Memory Along the Sequence

May 12, 2025 · 8 min · Duo An

MoE Parallelism Principles: GShard, Expert Parallel, and All-to-All

May 10, 2025 · 8 min · Duo An

April  5

ZeRO: Partitioning Optimizer State, Gradients, and Parameters

April 27, 2025 · 9 min · Duo An

Megatron Internals III: Mixed Precision, Loss Scaling, and Grad Clipping

April 26, 2025 · 8 min · Duo An

Megatron Internals II: Column/Row Parallel Linear and Vocab Parallel Embedding

April 19, 2025 · 7 min · Duo An

Megatron Internals I: Building the DP / TP / PP Process Groups

April 12, 2025 · 8 min · Duo An

Data Parallelism: From Parameter Server to Ring All-Reduce

April 6, 2025 · 9 min · Duo An

March  2

Pipeline Parallelism from First Principles: Why GPipe Split the Batch

March 16, 2025 · 8 min · Duo An

LLM Training Series: A Systems Map for Distributed Transformers

March 2, 2025 · 8 min · Duo An

February  1

From Scaling Laws to Cluster Size: Capacity Planning That Survives Contact With GPUs

February 2, 2025 · 4 min · Duo An