#评测/基准

Muse Spark 1.2 places Meta on the Cost per Task Pareto frontier, scoring 6 points below Claude Opus 5 at ~1/6th of the cost At Meta's $1.25/$4.25 per 1M token pricing, Muse Spark 1.2 (xhigh) sits on the Pareto frontier of Intelligence Index vs Cost per Task. It delivers comparable intelligence to Claude Opus 4.8 (max, $2.03) at a fifth of the cost per task, and undercuts GPT-5.6 Sol (high, $0.55), GPT-5.6 Terra (max, $0.61), and Kimi K3 (max, $0.87). The nearest cheaper options are Grok 4.5 (high, $0.36) and GPT-5.6 Sol (medium, $0.37), and they all sit below it on the Index. The step up from Muse Spark 1.1 ($0.29 per task) comes at unchanged per-token pricing, with the increase driven by heavier token usage on agentic tasks.

Every agent memory system publishes its own numbers. Almost none of them are comparable. Different datasets, different answer models, different judges. When a score moves, you can’t tell whether the memory got better or the setup got kinder. The Agent Memory Challenge has been open since Wednesday. Submissions close August 7. The design, briefly: You expose two endpoints — add and search. That’s the entire integration. Answer generation, judging, aggregation and orchestration run on our side, identically for every entry, and we pay for that compute. Your internals stay closed. Text track 10+ datasets consolidated (PersonaMem, LoCoMo-Refined, CLBench, BEAM, LongMemEval, ScriptMem and others). ~150M characters of history, ~5,000 questions, scored across seven capability dimensions rather than collapsed into one number. Code track — the part we think doesn’t exist yet: 12 repos, 150 base tasks, and 1,290 historical PRs annotated as strongly related, weakly related or unrelated to the task at hand. Strict temporal constraints, so nothing from the future leaks backward. It measures whether a system can retrieve and reuse engineering experience, not whether it can solve an issue cold. Two boards Open-source methods — ranked and eligible for rewards Commercial products — ranked separately for procurement reference Organized by 20+ universities and research institutions. Enter → https://agentmemories.ai/competition/ Protocol, submission guide, reference integrations → https://github.com/AML-memory/agent-memory-leaderboard Results go up mid-August. Logs, retrieved evidence and judge records stay public.

Alibaba's Qwen3.8 Max scores 56 on the Artificial Analysis Intelligence Index at $1.14 per task, but open weights leader Kimi K3 remains 1 point ahead at 25% lower cost per task ($0.86) @Alibaba_Qwen has released Qwen3.8 Max, which Alibaba states is a 2.4T total parameter MoE activating 95B parameters per forward pass. Alibaba has announced it plans to release the weights next week, a shift in strategy as it has typically kept its Max class of models proprietary. Once released, Qwen3.8 Max would be ~6x larger than Alibaba's largest open weights release to date (Qwen3.5 397B) and the second largest open weights model behind Kimi K3 (2.8T) Note: we earlier published results showing Qwen3.8 Max scoring 53 on the Artificial Analysis Intelligence Index. Those runs were affected by intermittent issues on the endpoint we were evaluating, and we have re-run all evaluations on Alibaba's public API endpoint Key takeaways: ➤ Qwen3.8 Max scores 56 on the Artificial Analysis Intelligence Index, up 10 points from Qwen3.7 Max (46). It is in line with Claude Opus 4.8 (max, 56) and sits second among labs from China, ahead of GLM-5.2 (max, 51) but behind Kimi K3 (max, 57) ➤ Qwen3.8 Max scores 1739 Elo on GDPval-AA, a 468 Elo gain over Qwen3.7 Max. This places it ahead of Kimi K3 (1685), effectively tied with Claude Fable 5 (1743) and GPT-5.6 Sol (max, 1730), and behind only Claude Opus 5 (max, 1852) ➤ Gains over Qwen3.7 Max span agentic evaluations, scientific reasoning and coding: Terminal-Bench v2.1 +6 points, CritPt +7 points, SciCode +4 points and HLE +3 points, with GPQA unchanged. AA-LCR (-2 points) and AA-Omniscience (-10 points, driven by a hallucination rate rising 23% to 40%) regress ➤ Qwen3.8 Max costs $1.14 per Intelligence Index task, more than double Qwen3.7 Max ($0.53), at ~1.3x Kimi K3 (max, $0.86) and ~2x GLM-5.2 (max, $0.57). Cost is driven in part by more turns on agentic evaluations, with GDPval-AA input tokens rising ~15x over Qwen3.7 Max, and output token usage up 45% to 145M ➤ The 𝜏³-Bench Banking result (42%) appears out of distribution. It is a 32 point gain over Qwen3.7 Max and places Qwen3.8 Max ahead of models that outscore it on other evaluations Key model details: ➤ Size: 2.4T total parameters, ~95B active per forward pass (MoE) ➤ Context window: 1M tokens ➤ Multimodal: text, image and video input with text output ➤ Pricing: $2.00/$6.00 per 1M input/output tokens on the @alibaba_cloud first-party API, with a $0.25 cache hit price. This is lower than Qwen3.7 Max across the board ($2.50/$7.50, with a $0.50 cache hit price) ➤ Availability: Alibaba Cloud first-party API. Alibaba states the weights will be released next week

qwen 3.8 max vs deepseek v4 flash 0731 vs kimi k3 vs gpt 5.6 sol – on rubik's cube and chess four frontier models built a rubik's cube stand and solved it, then built a chess board and played claude opus 5 on it the setup: @nousresearch's hermes agent cli on @openrouter tasks: 1. cube – build a 3d rubik's cube with a cli and a @threejs viewer, then solve an identical scrambled position on your own stand 2. chess – build a 3d chess stand, then play white against claude opus 5 as black, live, one move at a time. no engine, no solver, no opening book on either side. stockfish depth 14 grades every chess ply afterwards; neither player sees the score models: @deepseek_ai v4 flash 0731, @openai gpt-5.6 sol, @kimi_moonshot kimi k3, @alibaba_qwen qwen 3.8 max gpt-5.6 sol and deepseek v4 flash solved their cubes – sol in 24 moves and seventeen seconds, deepseek in 32. qwen and kimi never got there, giving up at 96 and 207 moves then all four built chess stands and played white against claude opus 5 on them, and all four resigned: deepseek on move 13, sol on 19, kimi on 21, qwen holding out longest at 29 - build time, both stands #1 gpt-5.6 sol – 16m 43s #2 deepseek v4 flash – 97m 39s #3 kimi k3 – 166m 09s #4 qwen 3.8 max – 215m 08s - build attempts before a working stand #1 gpt-5.6 sol – 3 #2 qwen 3.8 max – 4 #3 kimi k3 – 4 #4 deepseek v4 flash – 5 - total tokens #1 gpt-5.6 sol – 6,713,754 #2 qwen 3.8 max – 17,272,507 #3 kimi k3 – 22,427,504 #4 deepseek v4 flash – 27,417,442 - total price #1 deepseek v4 flash – $0.557 #2 gpt-5.6 sol – $6.319 #3 qwen 3.8 max – $10.270 #4 kimi k3 – $16.667 observations: • deepseek v4 flash is the cheapest model here by a margin nobody else is near, and it got there while being the least efficient of the four. it burned 27.4m tokens – more than anyone, 5m more than kimi – and still finished both benchmarks for $0.557. that is $0.02 per million tokens against kimi's $0.74. it also needed the most passes to produce working stands, five, and that did not matter: all five deepseek passes together cost a thirtieth of kimi's two • so what deepseek cannot do is get it right the first time. what it can do is get it right the fifth time, for half a dollar. that is a different thing to be buying – not a good first draft, but the option to keep asking • gpt-5.6 sol is the opposite profile and the strongest of the four on pure efficiency. 16m 43s to build both stands, 6.7m tokens, three passes – under 40% of the next lowest token count and a quarter of deepseek's, on an eighth of qwen's clock. it also solved the cube fastest of anyone, 24 moves in seventeen seconds. sol is what you reach for when you want the answer now and can absorb $0.94 per million • sol's weakness is in what it does not check. its chess viewer deleted the capturing piece instead of the captured one, so pieces disappeared off the board mid-game – a defect the fifty-cent deepseek stand did not have. fast and terse turns out to be the same dial as fast and unverified • qwen 3.8 max is not the cheap open-weights option it gets treated as. $10.270 across the two benchmarks, second most expensive of the four, 18x deepseek, and by a distance the slowest – 215 minutes of build time, nearly thirteen times sol's. what the money buys is judgment: it played eighteen moves without a single error worth a hundredth of a pawn, then made exactly one bad move in the whole game, and averaged 44.6 centipawns lost across the longest game any of the four managed. it also could not solve a rubik's cube in 96 tries • kimi k3 is the one line with no reading that flatters it. most expensive at $16.667, last on the cube at 207 moves, last at chess at 478 centipawns lost per move. it is also the model that verified hardest – on the cube it wrote its own integrity check instead of trusting its output. that makes the result worse rather than better: the checking was real, and the reasoning underneath it still was not follow @thehypedotnews for 24/7 ai news, analysis and breakdowns

Big news: Opus 5 (Max) is now #1 in the Fullstack Code Arena with 1,699 points! The Fullstack Leaderboard shows overall rankings across AI models on full-stack web development tasks: multi-step reasoning, tool use, and end-to-end app generation. Congrats again to the @AnthropicAI team on Opus 5 (Max)!

Exciting news: Qwen3.8-Max by @Alibaba_Qwen is #2 in Image-to-WebDev Arena! With 1,631 pts, it’s trailing only Claude Opus 5 (Max) by 39 pts! Congrats again to @Alibaba_Qwen on this huge release!

Qwen 3.8 Max beat Fable 5 at building 3D physics scenes for 7x cheaper! We gave two models the same task. Build three self-contained 3D scenes, each one HTML file with real physics that runs itself. Prompts: - A marble machine that lifts marbles up a wheel and drops them on a loop - A car factory assembly line - A sawmill cutting logs into planks Outputs: Qwen 3.8 Max: 46.6K tokens, $0.28 Fable 5: 38.7K tokens, $1.93 Qwen worked much harder on the details. Its textures and shadows look real. Fable 5 built a fast prototype that works and left it there. In the marble machine, Fable's marbles show up from nowhere at the top of the wheel. In Qwen's, you can see them go up. On the car line, Qwen's cars look much better and more real. Fable's sawmill is very strange. The saw looks wrong, and the log goes right through a beam before the saw cuts it. Atomic Chat will have day zero support to run Qwen 3.8 Max locally!

Qwen3.8-Max ranks #2 in Vision Arena scoring 1,305. Second only to Claude Fable 5 (High) which has only a 13pt lead.

DeepSeek V4-Flash isn’t just cheaper per token. It reportedly completes the same benchmark tasks as Fable 5 at 105× lower total cost, according to @ArtificialAnlys ! That's precisely why the Flash release is, for me, the DeepSeek 2.0 moment. It will cause a huge stir.