https://duoan.github.io/tags/paper-reading/ Recent content in Paper Reading on Duo's Tech Blog https://duoan.github.io/images/papermod-cover.png https://duoan.github.io/images/papermod-cover.png Hugo -- 0.153.1 en-us Mon, 13 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/argus-tracing-at-10000-gpu-scale/ Mon, 13 Jul 2026 00:00:00 +0000 https://duoan.github.io/posts/argus-tracing-at-10000-gpu-scale/ <h1 id="paper-reading-argus--always-on-tracing-at-10000-gpu-scale">Paper Reading: ARGUS — Always-On Tracing at 10,000+ GPU Scale</h1> <p>What Tencent built to catch fail-slow training jobs on 10k+ GPU clusters with under 2% overhead — plus Modal remasurements of always-on <code>torch.profiler</code> / <code>nsys</code> overhead and a tiny demo of the KDE + W₁ detection math.</p> <p><strong>Paper:</strong> <a href="https://arxiv.org/abs/2606.20374">ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters</a> (Zhou et al., Tencent, arXiv 2606.20374, submitted to ATC 2026)</p> <h2 id="tldr">TL;DR</h2> <p>Large LLM training jobs are synchronous: one slow rank, link, or host-side stall can waste thousands of GPU-hours without triggering a hard failure. Existing tools split into two camps:</p>