SiliconFlow· @SiliconFlowAI · X·· 2026-05-14AI 评分63
AI 导读
NousResearch发布了Token Superposition Training(TST),这是一种改进标准大语言模型预训练流程的方法。该技术无需改变模型架构、优化器、分词器或训练数据,即可在相同计算量(FLOPs)下实现2-3倍的训练时间加速。其核心是在训练的前三分之一阶段,让模型读取并预测连续的token包,对输入嵌入进行平均,并使用改进的交叉熵损失预测下一个token包;剩余训练时间则恢复为标准的下一个token预测。推理阶段的模型与传统预训练产生的模型完全相同。该方法已在270M、600M、3B的密集模型以及10B至1B的混合专家模型规模上得到验证。
正文 · 原文
bravooo
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.在 X 查看被引用的帖子
来源:SiliconFlow · x.com