#评测/基准

DeepSeek V4.1 Flash from @deepseek_ai on ARC-AGI (Verified): - ARC-AGI-2: 72.9%, $0.13/task - ARC-AGI-1: 94.5%, $0.07/task DeepSeek V4.1 Flash beats V4 Flash's best scores by 5.5 points on ARC-AGI-1 and 11.5 on ARC-AGI-2, but costs about 250% more per task.

We're launching our leaderboard for FrontierCode 1.1, a private evaluation from Cognition that tests models on completing coding tasks in a way that would actually get merged by maintainers of real-world repositories @cognition announced FrontierCode in June 2026 and updated it to version 1.1 in July. Cognition built the dataset with open source maintainers who spent more than 40 hours per task. Each task requires the agent to make a change to a real open source repository, then checks both unit tests and qualitative tests for what maintainers would require to merge the change. We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, using our Stirrup open source reference harness for apples-to-apples comparisons across models. All tasks are private to avoid contamination. Each attempt is graded with both blocking and non-blocking criteria. An attempt that fails any blocking check scores zero, and one that passes them all scores between 70% and 100%. Unlike other coding evaluations, FrontierCode has a more holistic view of success based on real usage. Models need to pass unit tests, but are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them. Key takeaways: ➤ Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at a score of 52% after passing blockers in 58% of its attempts. Nine of the 100 tasks were not passed in any attempt by any of the models we evaluated for launch ➤ The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44% ➤ GPT-6.1 Sol (max) offers the strongest performance for its cost: At $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback). This means it achieves similar performance to leading Anthropic models, at less than a tenth of the cost per task Thank you to @silasalberti, @ybenpan, @cl571128, and the rest of the team at Cognition for building FrontierCode and collaborating with us to bring it to Artificial Analysis - we’re working to add FrontierCode to our coding agent leaderboard very soon.

Meta Muse's retention is out, and it's almost as good as ChatGPT! It's higher than ChatGPT at D7 (>30%) and just lower at D14 (22%). Not quite in the elite D14 tier of WA (67%) YT (59%) Insta (54%) or TikTok/Snap (37%), though. Muse is one of the highest retaining AI products!

We're launching our leaderboard for FrontierCode 1.1, a private evaluation from Cognition that tests models on completing coding tasks in a way that would actually get merged by maintainers of real-world repositories @cognition announced FrontierCode in June 2026 and updated it to version 1.1 in July. Cognition built the dataset with open source maintainers who spent more than 40 hours per task. Each task requires the agent to make a change to a real open source repository, then checks both unit tests and qualitative tests for what maintainers would require to merge the change. We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, using our Stirrup open source reference harness for apples-to-apples comparisons across models. All tasks are private to avoid contamination. Each attempt is graded with both blocking and non-blocking criteria. An attempt that fails any blocking check scores zero, and one that passes them all scores between 70% and 100%. Unlike other coding evaluations, FrontierCode has a more holistic view of success based on real usage. Models need to pass unit tests, but are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them. Key takeaways: ➤ Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at a score of 52% after passing blockers in 58% of its attempts. Nine of the 100 tasks were not passed in any attempt by any of the models we evaluated for launch ➤ The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44% ➤ GPT-6.1 Sol (max) offers the strongest performance for its cost: At $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback). This means it achieves similar performance to leading Anthropic models, at less than a tenth of the cost per task Thank you to @silasalberti, @ybenpan, @cl571128, and the rest of the team at Cognition for building FrontierCode and collaborating with us to bring it to Artificial Analysis - we’re working to add FrontierCode to our coding agent leaderboard very soon.

introducing muse wonder our new frontier model. state-of-the-art at personal agents and professional agents. our largest and most capable model yet—rivals the big bois

我们很高兴推出 Muse Wonder 系列，包括 Muse Wonder 和 Muse Wonder Full，这是我们迄今为止最强大的模型。两者都是前沿模型，旨在为个人和专业代理任务提供最先进的性能。 Muse Wonder 今天可通过 Meta Model API 和 Muse Code 使用。Muse Wonder Full 今天面向 Muse Code 订阅者提供。Muse Wonder 很快还将为我们的个人代理产品 Muse 中的体验提供支持，首先向订阅者推出。

Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.

User: For a design agency, complete 3D SVG iconography of 100+ icons. Ling-3.1-flash: 1,000+ shapes GPT-5.6 Sol: under 200 Ling-3.1-flash is the new model from @AntLingAGI, and we found three massive strengths from their model in our blind votes: detailed SVGs, animated SVGs, and entire mobile apps. Full film below! Try out http://designarena.ai.

Introducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality. We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.

New ARC Prize 2026 - ARC-AGI-2 High Score 88.06% by @tufalabs The $150K Bonus Prize, on top of the guaranteed prizes, will be split among all teams scoring over 85%

认识一下 Mistral Large 4，又名 Le Chonk。 • 1T 参数，原生多模态。49B 激活。 在综合基准测试中，它是来自美国或欧洲的最佳开放权重模型。 • 在关键工作负载上达到最先进水平，包括网络防御、制造业和金融，并且在视觉定位方面超越了闭源前沿模型。 • 端到端在欧洲打造，并可通过我们自己的 Mistral Cloud 基础设施从欧洲部署。 • 今天即可通过 API 向所有人开放。正与网络安全合作伙伴私下合作。 开放权重版本将于十月底发布。

New Research: We are releasing ExplorationBench, a benchmark for measuring how AI systems explore. Scientific discovery begins where known problems end: a system has to frame hypotheses, design experiments, and learn from the results. Evaluating this is hard. Genuinely new answers cannot be checked quickly, and in familiar domains a model can simply recall what it has seen. Addressing this challenge, researchers from Tencent Hy, Fudan University, and Tsinghua University built verifiable Alien Worlds. Their rules are executable, so every answer is checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. 🔹 Two sandboxes: AlienCode (31 hidden rule changes, 70 tasks) and AlienLogic (24 patched inference rules, 70 theorems) 🔹 A flawed manual, four rounds of self-designed probes, and closed-book tests after every round 🔹 Every answer graded by an interpreter or a proof checker, with no LLM judge What we found across 10 frontier AI systems: 1️⃣ Getting feedback is more effective than thinking alone. No AlienCode run starts above 15.7%; after four rounds the best reaches 89.0%, while the same turns without feedback stay at 0.5–11.0%. 2️⃣ Designing the experiments matters. Replaying a system's own best probes gives it exactly the same evidence, yet in AlienCode 9 of 10 systems do worse than when they chose the probes themselves. 3️⃣ Knowing a rule is not using it. Even when every required rule is stated correctly, tasks are solved only 73.4% of the time. 4️⃣ One score hides a lot. The same system under the same budget ended anywhere from 5.7% to 79.0%, and rankings barely transfer between the two worlds. CL-bench asked whether models can learn from context. ExplorationBench asks whether they can discover the rules themselves. 📄 Paper: https://arxiv.org/abs/2609.30199 🌐 Website & leaderboard: https://explorationbench.com 📝 Blog: https://explorationbench.com/blog/ 💻 Code (coming soon): https://github.com/Tencent-Hunyuan/ExplorationBench

We gave 8 SoTA video world models a physics exam. Top score: 57.76 / 100. One revealing result: free-fall video passed the consistency screen. Their average composite score was just 26.61/100. Physics is still undefeated. Today we introduce World Models' Last Exam in Physics — 40 controlled tasks, 9 physical categories, measured against real physical laws. Full breakdown ⬇️ Github: https://github.com/Einsia/phys-last-exam Project: https://lab.einsia.ai/phys-last-exam Paper: https://arxiv.org/pdf/2610.08791

What began as a UC Berkeley research project is becoming the infrastructure the world relies on to understand and advance AI. Today, @arena announced a $200 million Series B at a $3.1 billion valuation and launched the Arena AI Alignment Index, a new way to measure how AI agents behave in the real world. We at @felicis first backed Arena at the seed and led its Series A because @ml_angelopoulos, @infwinston, and @istoica05 understood the ground truth in AI: you can’t improve what you can’t measure. Congratulations to the entire Arena team on this milestone and the important work ahead! Read Arena’s announcement: https://arena.ai/blog/series-b And learn how Anastasios and Wei-Lin turned a weekend research project into AI’s trusted evaluation platform: http://felicis.link/2Eouv0v

隆重推出 Claude Haiku 5.5：我们迄今为止发布的最便宜、最快、最强大的小型模型。 平均而言，它的运行成本比 Claude Haiku 4.5 低约 75%。