Martian 推出 AI Frontier，一个按任务、质量、实际成本和可靠性比较 LLM，并展示组合不同模型效果的仪表盘。在 16 个基准（TerminalBench、LiveCodeBench 等）上，其 oracle 路由在同成本下平均降低 54% 误差，或以低 85% 的 API 成本达到各基准最优模型的质量；引用数据称误差比单一最优 LLM 少 46%。
Picking one "best model" is starting to look like the wrong unit of optimization.
@withmartian just released AI Frontier, a dashboard for comparing LLMs by task, quality, actual cost, and reliability, then seeing whether combining different models gives better results.
Across 16 benchmarks, Martian’s oracle routing cut average error 54% at matched cost, or matched each benchmark’s top-model quality at 85% lower API cost.
The idea is that there may be no single "best" LLM: Different models perform better on different workloads, so AI Frontier shows when choosing or combining models can produce a better mix of quality, cost, and reliability than using one model for everything.
Output length, reasoning behavior, retries, and reliability all affect what you eventually pay to get a usable answer.
Martian is also measuring how consistently models solve the same kinds of problems, which makes the cost picture more useful. A cheap model that needs several attempts may not be cheap at the system level.
Model economics should probably be measured as cost per acceptable result, not simply dollars per million tokens.
We got 46% fewer errors than the single best LLM across the 16 most used benchmarks (TerminalBench, LiveCodeBench, etc). Here's how that's possible and what each model can achieve when used optimally (every benchmarks misses the majority of model capabilities) 👇 Interactive Site: https://aifrontier.withmartian.com/ Academic Paper: https://arxiv.org/abs/2606.26836在 X 查看被引用的帖子
来源：Rohan Paul · x.com