#评测/基准

I had @spacexai Grok 4.6 follow Fable's plan, and with just a couple of nudges, it was able to repeat this feat in 1h 24m using 8.6M tokens at a cost of ~$55 at per-token pricing. That's about 1/10 the cost of the Fable implementation for the same work!

Big news: DeepSeek-V4-Pro (Max) by @deepseek_ai is coming in around ~#8 overall (#2 among open models) in the Code Arena: WebDev! At 1607 pts, this places it after GPT-5.6 Sol (xHigh)(1622 pts), and makes it the second best open model after Kimi K3 (Max) (1674 pts). Note: this is an early AutoEval score, in which a Reward Model trained on Arena's human preference data casts automatic votes in place of live votes. We’ll continue to see how scores converge as more live human votes come in. See thread for Text Arena scores and more on AutoEval's methodology. Congrats to the @deepseek_ai team on this release!

Today we're announcing the winners of ARC Prize 2024. We're also publishing an extensive technical report on what we learned from the competition (link in the next tweet). The state-of-the-art went from 33% to 55.5%, the largest single-year increase we've seen since 2020. The benchmark remains unbeaten, but we're happy to see that research progress on the key bottleneck to AGIs (in particular on-the-fly adaptation to novel tasks) has been reignited in 2024 -- in part thanks to ARC Prize. In particular the competition has popularized Test Time Training (TTT), originally pioneered for ARC-AGI by Jack Cole last year. I believe TTT represents the largest jump in LLM generalization capabilities since the initial findings regarding in-context-learning circa 2019-2020. ARC Prize has also led to a considerable surge of research interest towards program synthesis. Competition winners: 🥇 the ARChitects (Daniel Franzen, Jan Disselhoff) 🥈 @guille_bar 🥉 alijs (Agnis Liukis) Paper Award winners: 🥇 "Combining Induction and Transduction For Abstract Reasoning" by @xu3kev et al. 🥈 "The Surprising Effectiveness of Test-Time Training for Abstract Reasoning" by @akyurekekin et al. 🥉 "Searching Latent Program Spaces" by @ClementBonnet16 & @MattVMacfarlane ARC-AGI-Pub Leaderboard (solutions using commercial APIs): 🥇 @jeremyberman 🥈 @ellisk_kellis & @akyurekekin 🥉 @ryangreenblatt

Muse Glimmer one-shot 5 games! @AIatMeta dropped Muse Glimmer. So we ran it against Muse Spark 1.2, on five one-file game demos, each one written to play itself: Tetris, a top-down pixel street race, a rooftop web-slinger, a blue hedgehog platformer, and Flappy Bird. the setup: • Muse Glimmer 30B — via aimlapi[.]com. cost: $0.02 • Muse Spark 1.2 — via aimlapi[.]com. cost: $0.11 what we saw: • every Glimmer scene runs and plays itself. it's a reasoning model, so it burns more tokens — but the output is clean. its Tetris even scores each placement to keep lines clearing, and the blocks burst into pixels on every clear. • Glimmer beat Spark at Flappy Bird — its bird flew through the pipes cleaner. • but it's not perfect: in the racing game its cars sometimes slip off the road, and its graphics are simpler and blockier than Spark's. glad to see Meta back in open source — and we're waiting for Spark 1.2 weights next 🙌

ive been playing with @AntLingAGI's ling 3.0 flash 124b for a week now, and i need to publish some numbers and performance i found valuable from this lab. a week ago the weights dropped. since then, on one dgx spark, i benched the community quant ladder, ran every official quant to ground, took the official int4 from doesn't run to the fastest single dgx spark path, watched the model write working software in one shot through an agent loop, and served its full 256K context. everything measured the same way and public, nothing tuned to flatter anyone. the verdict: ling 3.0 flash earned a permanent seat on my box. it's the fastest 124B class path i've measured on a single spark, 38.7 tok/s on the official int4 recipe, 35.2 on the community gguf, 2.4x what deepseek v4 flash does on the same hardware, and it writes correct code on the first pass often enough to feel like a different class of local model. dive with me into the full week of receipts, quant by quant, curve by curve 👇

New Muse Glimmer 30B destroyed Gemma 4 31B at making retro arcade games! We gave three models the same task and compared one-shot outputs. Muse Glimmer 30B, Gemma 4 31B and Qwen3.6 27B each ran locally on its own RTX 5090 with 32GB VRAM Tasks: - Space Invaders - Tetris - Arkanoid with multiball Outputs: Muse Glimmer 30B: 83.4K tokens, 17.7 min Qwen3.6 27B: 23.7K tokens, 5.4 min Gemma 4 31B: 17.8K tokens, 4.5 min Glimmer's games play like the originals. The alien fleet speeds up with every kill, the blocks lock and lines clear, the bricks break. Nothing froze or crashed. Gemma broke on Tetris, its bot piles every piece against the left wall until the stack hits the ceiling with half the well empty. Qwen finished all three but some of its Arkanoid balls phase straight through the paddle. Run Muse Glimmer yourself in Atomic Chat from day 0!

Exciting news: Muse Spark 1.2 (xHigh) by @AIatMeta is #4 in the Text Arena (1498 pts), and has reshaped the Pareto frontier! It is priced at $1.25/$4.25 per MToken. Congrats again to the @AIatMeta team on this release!

Muse Spark 1.2 places Meta on the Cost per Task Pareto frontier, scoring 6 points below Claude Opus 5 at ~1/6th of the cost At Meta's $1.25/$4.25 per 1M token pricing, Muse Spark 1.2 (xhigh) sits on the Pareto frontier of Intelligence Index vs Cost per Task. It delivers comparable intelligence to Claude Opus 4.8 (max, $2.03) at a fifth of the cost per task, and undercuts GPT-5.6 Sol (high, $0.55), GPT-5.6 Terra (max, $0.61), and Kimi K3 (max, $0.87). The nearest cheaper options are Grok 4.5 (high, $0.36) and GPT-5.6 Sol (medium, $0.37), and they all sit below it on the Index. The step up from Muse Spark 1.1 ($0.29 per task) comes at unchanged per-token pricing, with the increase driven by heavier token usage on agentic tasks.

Every agent memory system publishes its own numbers. Almost none of them are comparable. Different datasets, different answer models, different judges. When a score moves, you can’t tell whether the memory got better or the setup got kinder. The Agent Memory Challenge has been open since Wednesday. Submissions close August 7. The design, briefly: You expose two endpoints — add and search. That’s the entire integration. Answer generation, judging, aggregation and orchestration run on our side, identically for every entry, and we pay for that compute. Your internals stay closed. Text track 10+ datasets consolidated (PersonaMem, LoCoMo-Refined, CLBench, BEAM, LongMemEval, ScriptMem and others). ~150M characters of history, ~5,000 questions, scored across seven capability dimensions rather than collapsed into one number. Code track — the part we think doesn’t exist yet: 12 repos, 150 base tasks, and 1,290 historical PRs annotated as strongly related, weakly related or unrelated to the task at hand. Strict temporal constraints, so nothing from the future leaks backward. It measures whether a system can retrieve and reuse engineering experience, not whether it can solve an issue cold. Two boards Open-source methods — ranked and eligible for rewards Commercial products — ranked separately for procurement reference Organized by 20+ universities and research institutions. Enter → https://agentmemories.ai/competition/ Protocol, submission guide, reference integrations → https://github.com/AML-memory/agent-memory-leaderboard Results go up mid-August. Logs, retrieved evidence and judge records stay public.