#评测/基准

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

Does your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge: 1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost 2️⃣A simple harness can be competitive 3️⃣The native harness isn’t always the best. Millions of people are using coding agents, but the impact of harness choice remains unclear. (1/n) More details in the thread. 🧵

Exciting news: DeepSeek-V4.1-Flash (Max) by @deepseek_ai just landed in Agent Arena at #3 among open models! With +4.87% net improvement and a median cost per task of $0.07 it reshaped the Pareto frontier. Among the top 3 open models, DeepSeek-V4.1-Flash (Max) has the lowest median cost per task. Its +4.87% net improvement is within 0.09 percentage points of Hy4 preview (ranked #2) at 68% lower cost, and within 1.52 percentage points of Kimi K3 (Max) (ranked #1) at 91% lower cost. - Kimi K3 (Max): +6.39% | $0.77/task - Hy4 preview: +4.96% | $0.22/task - DeepSeek-V4.1-Flash (Max): +4.87% | $0.07/task See the full Pareto Frontier below. DeepSeek-V4.1-Flash (Max) is ranked #12 overall, and by signal landed #4 Confirmed Success with +13.75%! Congrats to the @deepseek_ai team on this release!

🚀 推出 DeepSeek-V4.1-Flash：更智能、更快速、更高效。 🔹 推出我们新架构家族中最小的模型，具备原生视觉理解能力。 🔹 旨在实现更强大的能力、更快的推理、更高的吞吐量，并可扩展至更大的模型。 1/6

For the next 72 hours on Arena, you can use GPT-Image-2.5 Sunburst directly at no cost. @OpenAI’s latest image model was just ranked #1 in the Text-to-Image Arena, the Image Edit Arena, and in the Multi-Image Edit Arena. Find the link below, select the model in Direct Mode, and test it directly for yourself! Direct Mode lets users directly select models by name. Access to GPT-Image-2.5 Sunburst ends Sept. 13 at 8 a.m. PT. Afterward, it will remain available anonymously in Battle and Agent Mode. User votes from Battle and Agent Mode are used to power our real-world leaderboard rankings. In these modes, models are always anonymized.

We've never seen this before. The biggest jump in Vending-Bench history. GPT-6 Astra is better at making money and more ethical than Claude Fable 5.1. Surprising, because: 1. First time ever that OpenAI is #1 on Vending-Bench 2. The best model is no longer the unethical one.

Muse Spark 1.3 Max by @AIatMeta has reshaped the Pareto frontier for Code Arena: WebDev! Meta's latest model at Max reasoning is doing something interesting on the Arena Pareto frontier: it's the only model holding down the wide price band between Qwen3.8-max ($5/MToken) and Qwen3.8-Flash-Next ($0.39/MToken). The next closest model to the frontier is Meta's own, Muse Spark 1.3 on xHigh reasoning. Muse Spark 1.3 Max performs just 20 pts lower than Qwen3.8 (Max) while costing 30% less, and 24 pts from Kimi K3 (Max) while costing 70% less. - Muse Spark 1.3 Max: 1650 pts | $3.50/M - Qwen3.8 (Max): 1670 pts | $5/M - Kimi K3 (Max): 1674 pts | $12/M Overall in Code Arena: WebDev, Muse Spark 1.3 Max landed #8, ahead of Claude Fable 5 at #10 (+22 pts), Grok-4.6 (High) at #12 (+26 pts), and GPT-5.6 Sol at #14 (+33 pts). Congrats to the @AIatMeta team on this release!