Posts

Most “Scalable” Multimodal Pipelines Don’t Survive Foundation-Model Scale A lot of multimodal pipelines claim to scale. In practice, they often depend on at least one of the following: global shuffles (groupBy/join/repartition), materializing massive intermediate datasets, centralized coordination that becomes a bottleneck, or brittle recovery logic (rerun-the-world on failure). That works for demos. It breaks at foundation-model scale. This series is about a different design point: A streaming-first multimodal pipeline that scales linearly with data and hardware — with no global shuffle, and resumable at partition granularity. ...

SiQ-VL: A Curriculum for Small VLMs When Compute Is the Hard Constraint Most VLM writeups assume a cluster. SiQ-VL started from the opposite constraint: one (or few) GPUs, and the question was which design choices still buy capability when you cannot buy FLOPs. This post is the consolidated field guide for that project — architecture, token economics, staged training, and offline Chain-of-Thought (CoT) distillation — replacing three earlier notes that said the same thing three ways. Kernel-level throughput (how we pushed Stage-1 from ~15K to ~100K real tokens/s on Blackwell) lives in the companion post: Optimizing VLM Training on One GPU. ...

From Scaling Laws to Cluster Size: Capacity Planning That Survives Contact With GPUs Scaling laws tell you how loss should move with tokens and parameters. Clusters tell you what you can actually buy. Capacity planning is the bridge — and most teams skip it until the first failed launch burns a week of calendar time and a pile of GPU-hours. This post is a reproducible planning pipeline: tokens → steps → FLOPs → GPU-hours → forced parallelism. The arithmetic lives in playground/capacity_plan.py; the figures below are generated from its JSON output. Not exact. Accurate enough to catch 10× fantasy plans. ...