Yann LeCun· @ylecun · X·· 2025-07-12
AI 导读
Micah Goldblum 指出，batch size 为 1 的无动量 vanilla SGD（入门 ML 的首个优化器）在 LLM 预训练中，per-FLOP 速度几乎与 AdamW 相当。
正文 · 原文
The optimal batch size is 1
(For suitable definitions of "optimal")
🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? 📜 1/n https://t.co/pE24dfjjlK在 X 查看被引用的帖子
来源：Yann LeCun · x.com