#评测/基准

What began as a UC Berkeley research project is becoming the infrastructure the world relies on to understand and advance AI. Today, @arena announced a $200 million Series B at a $3.1 billion valuation and launched the Arena AI Alignment Index, a new way to measure how AI agents behave in the real world. We at @felicis first backed Arena at the seed and led its Series A because @ml_angelopoulos, @infwinston, and @istoica05 understood the ground truth in AI: you can’t improve what you can’t measure. Congratulations to the entire Arena team on this milestone and the important work ahead! Read Arena’s announcement: https://arena.ai/blog/series-b And learn how Anastasios and Wei-Lin turned a weekend research project into AI’s trusted evaluation platform: http://felicis.link/2Eouv0v

The seventh edition of our Top 100 Consumer AI Apps is here. New this time: a revenue leaderboard, alongside the usual web and mobile traffic rankings. Three years ago we published the first edition. ChatGPT was #1, Claude was unranked, and the entire category was chatbots, image generators, and not much else. In today's edition: - ChatGPT still holds the throne, now with 1B+ monthly actives on mobile - Claude has climbed to #3 on web with nearly 1B monthly visits - The category has expanded to vibe coding (Lovable, Cursor, Replit), music (Suno), design (Figma), voice (ElevenLabs), video (Higgsfield, Kling), agents (Manus), and even hardware (Plaud) Full breakdown from @omooretweets: https://www.a16z.news/p/top-100-consumer-ai-apps-seventh

The seventh edition of our Top 100 Consumer AI Apps is here. New this time: a revenue leaderboard, alongside the usual web and mobile traffic rankings. Three years ago we published the first edition. ChatGPT was #1, Claude was unranked, and the entire category was chatbots, image generators, and not much else. In today's edition: - ChatGPT still holds the throne, now with 1B+ monthly actives on mobile - Claude has climbed to #3 on web with nearly 1B monthly visits - The category has expanded to vibe coding (Lovable, Cursor, Replit), music (Suno), design (Figma), voice (ElevenLabs), video (Higgsfield, Kling), agents (Manus), and even hardware (Plaud) Full breakdown from @omooretweets: https://www.a16z.news/p/top-100-consumer-ai-apps-seventh

隆重推出 Claude Opus 5.5，这是我们全新的 Claude 5.5 系列中的首个模型。 在大多数任务上，它的表现达到了 Claude Fable 5.1 的水平，而运行成本比 Opus 5 低 40%。

Today we're announcing the winners of ARC Prize 2024. We're also publishing an extensive technical report on what we learned from the competition (link in the next tweet). The state-of-the-art went from 33% to 55.5%, the largest single-year increase we've seen since 2020. The benchmark remains unbeaten, but we're happy to see that research progress on the key bottleneck to AGIs (in particular on-the-fly adaptation to novel tasks) has been reignited in 2024 -- in part thanks to ARC Prize. In particular the competition has popularized Test Time Training (TTT), originally pioneered for ARC-AGI by Jack Cole last year. I believe TTT represents the largest jump in LLM generalization capabilities since the initial findings regarding in-context-learning circa 2019-2020. ARC Prize has also led to a considerable surge of research interest towards program synthesis. Competition winners: 🥇 the ARChitects (Daniel Franzen, Jan Disselhoff) 🥈 @guille_bar 🥉 alijs (Agnis Liukis) Paper Award winners: 🥇 "Combining Induction and Transduction For Abstract Reasoning" by @xu3kev et al. 🥈 "The Surprising Effectiveness of Test-Time Training for Abstract Reasoning" by @akyurekekin et al. 🥉 "Searching Latent Program Spaces" by @ClementBonnet16 & @MattVMacfarlane ARC-AGI-Pub Leaderboard (solutions using commercial APIs): 🥇 @jeremyberman 🥈 @ellisk_kellis & @akyurekekin 🥉 @ryangreenblatt

Every agent memory system publishes its own numbers. Almost none of them are comparable. Different datasets, different answer models, different judges. When a score moves, you can’t tell whether the memory got better or the setup got kinder. The Agent Memory Challenge has been open since Wednesday. Submissions close August 7. The design, briefly: You expose two endpoints — add and search. That’s the entire integration. Answer generation, judging, aggregation and orchestration run on our side, identically for every entry, and we pay for that compute. Your internals stay closed. Text track 10+ datasets consolidated (PersonaMem, LoCoMo-Refined, CLBench, BEAM, LongMemEval, ScriptMem and others). ~150M characters of history, ~5,000 questions, scored across seven capability dimensions rather than collapsed into one number. Code track — the part we think doesn’t exist yet: 12 repos, 150 base tasks, and 1,290 historical PRs annotated as strongly related, weakly related or unrelated to the task at hand. Strict temporal constraints, so nothing from the future leaks backward. It measures whether a system can retrieve and reuse engineering experience, not whether it can solve an issue cold. Two boards Open-source methods — ranked and eligible for rewards Commercial products — ranked separately for procurement reference Organized by 20+ universities and research institutions. Enter → https://agentmemories.ai/competition/ Protocol, submission guide, reference integrations → https://github.com/AML-memory/agent-memory-leaderboard Results go up mid-August. Logs, retrieved evidence and judge records stay public.

Exciting news: Qwen3.8-Max by @Alibaba_Qwen is #2 in Image-to-WebDev Arena! With 1,631 pts, it’s trailing only Claude Opus 5 (Max) by 39 pts! Congrats again to @Alibaba_Qwen on this huge release!

Qwen3.8-Max ranks #2 in Vision Arena scoring 1,305. Second only to Claude Fable 5 (High) which has only a 13pt lead.

MiniMax H3 by @MiniMax_AI is in the Video Arena! Available in Text-To-Video and Image-To-Video arenas. Open weights coming soon. Bring your most creative prompts and start voting. Scores on the way.

We've updated FrontierCode 1.1 to reflect new discounts for GPT-5.6 Terra and GPT-5.6 Luna. With these new costs, the GPT-5.6 series sits on the pareto curve of price/performance efficiency.

Code Arena now measures fullstack capabilities! View overall rankings across AI models on full-stack web development tasks: multi-step reasoning, tool use, and end-to-end app generation. - Kimi K3 (Max) takes #1 - GPT 5.6 Sol (xHigh) at #2 - Claude Fable 5 at #3 See more scores at: http://arena.ai/leaderboard/code/webdev/fullstack

BREAKING - OFFICIAL RESULTS: GPT-5.6 Sol by @OpenAI is 1st overall on Design Arena with an Elo of 1353. This puts GPT-5.6 Sol above Claude Fable 5 by @AnthropicAI and in the same performance band as GLM 5.2 by @Zai_org on frontend design. This is an 18-position and 60-point Elo leap from GPT-5.5. GPT-5.6 Sol also establishes a new Pareto frontier for preference vs. speed, faster than any model at this performance. Congratulations to the @OpenAI team on the launch!

SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination. Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails. Key takeaways: ➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks ➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49) ➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing ➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh) ➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5) ➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%) Congratulations to @SpaceXAI and @elonmusk on topping the leaderboard!

Grok 4.5 is #1 on Harvey's Legal Agent Benchmark