https://x.com/i/article/2101066086867492864

Is Astra more frustrating to use than Fable? The results are in.

by @DawidGalarowicz

Boris Cherny has shared that the Claude Code team tracks profanities across user messages to understand product experience. We took that idea further - what can praise and complaints tell us about coding agents themselves?

Having assessed 20,840 traces from Agent Arena: Code between August 1 - September 13 across 29 models, we find that direct feedback unlocks novel opportunities in tracking the frontier.

Key findings

- Today’s frontier models receive markedly more positive feedback. Fable 5.1 and Astra improve on their predecessors by 23.7 and 13.3 points on Arena’s praise/complaint score. Overall, newer models across labs tend to show a more positive feedback balance.

- Broken code is still the biggest driver of complaints, sloppy behavior comes second. Across all complaint-bearing traces, 69.1% point to code not working. The next themes are incomplete output (27.1%), weak finish and usability (27.1%), and ignored instructions (20.3%). Given that a trace can contain multiple complaints, the analysis assigned all applicable labels.

- Complaints are starting to shift toward how the agents do the work. More recent models receive fewer complaints than their predecessors, and the complaints are less focused on code simply not working (e.g., 70.9% for Fable 5 versus 58.3% for Fable 5.1). For example, users flag issues with the polish and documentation of the outputs.

- Models share common weaknesses, but differ in how they disappoint. Fable 5.1 attracts fewer “slop” and design complaints than Astra (3.9% versus 5.7% of sampled traces), and fewer complaints about being “flaky” (3.1% versus 5.0%). This reverses in how efficient the models are.

Developers feel the upgrades

Arena today generates a praise/complaint score per model - whether the task-level feedback contains more praise (”I like this design”) or more complaints (e.g., “the bug is still there, you didn’t fix it as planned”). The evaluation is computed across conversations within Agent Mode, where an agent is randomly selected for a user, and allows for creations of Agent Arena that track improvements on the baseline.

The most recent Agent Arena: Code leaderboard shows a striking gap between the recent frontier models from OpenAI and Anthropic, and their previous generations. Claude Fable 5.1 scores 41.58%, representing a 23.7-point uplift over Fable 5. Similarly, GPT 6 Astra reaches 37.53%, a 13.3-point uplift over GPT 5.6 Sol. The numbers show a meaningful change in how developers feel about their agents.

The capability acceleration is especially clear when previous iterations are considered. For example, the score for Opus 5 rose from 17.66 to 20.04 versus Opus 4.8. Additionally, being newer was not a guarantee, as GLM 5.3 sits below GLM 5.2 in the ranking. On average, model iterations led to incremental wins, rather than step-function changes we are seeing now.

Users still struggle to get the code that works

Most importantly, this data allow us to break the user messages up into 15 unique drivers of feedback. These characteristics broadly fall into two categories - result quality and agent behavior.

What is clear is that despite the frontier moving forward, the key problem for AI users is simply getting back code that works. This problem appeared in 69.1% of complaint-bearing traces, including comments on both runs bugging out as well as the results being wrong. The next area relates to “taste” - delivering high quality design (27.1% of traces including complaints). This covers finish, presentation and usability. Here, users call out “slop”. Finally, doing the work fully (27.1%) is the third common problem, an agent trait often called “laziness”.

At the bottom of the priority list, there are spaces such as Taking initiative (1.5%), Architecting well (1.7%) and Sharing good ideas (2.4%). This reveals the hierarchy of needs - before an agent can become a collaborator, developers need a reliable implementer.

Fable and Astra frustrate differently

Using our data we can also confirm three commonly held views about the current frontier.

The results show Fable 5.1 is much better at front-end than OpenAI’s Astra - 3.9% of all traces related to that, compared to 5.7% for the competitor. It also underscores how competitive Chinese models are at design, with Hy4 Preview and DeepSeek V4 Pro in Astra’s neighborhood.

Astra being “flaky” and going off-piste more often then Fable is confirmed here too. Astra ranked only fifth on instruction following, and the top three spots were claimed by Gemini 3.8 Flash, Fable 5.1 and Hy4 Preview.

Lastly, the new Fable’s communication style beats competitors based on direct user feedback. Model responses being more understandable was a big feature of this release. This area is also where the DeepSeek’s model excelled.

As for Astra’s advantages, the key one is working efficiently. Complaints about speed, wasted work or unnecessary back-and-forth appear in 1% of sampled traces for Astra, compared with 2.4% for Fable. In fact, Fable 5.1 is the worst out of all deployments shown.

Complaints are moving from “if it works” to “how it is done”

A deeper dive into the frontier also shows that as more capable models get released, fewer traces contain complaints, and the complaints move beyond agents writing code that does not compile.

For example, users of Fable 5 (High) complained in 29.9% of traces, compared with 18.7% for Fable 5.1 (Max). Among complaint-bearing traces, the share mentioning working results fell from 70.9% to 58.3%. Complaints therefore became less concentrated on whether the result worked.

This points to the balance shifting from developer-AI collaboration being about generating scripts that hopefully run to agents delivering software that human can iterate on, and be opinionated about.

Methodology

Data

We analyzed 20,840 Agent Mode code traces. A trace comprises a session/model pair, including its branches. The sample covered August 1 - September 13, 2026. Total of 29 model/reasoning deployments were analyzed in creation of this article.

Approach

We used LLM-as-a-judge to extract praise and complaints from user messages, retaining source quotations and assigning feedback to 15 shared driver categories. Categories and sentiments can overlap. Where shown, 95% intervals are used. The driver set was established after clustering user feedback and manually reviewing traces to obtain a parsimonious and a widely-applicable group.

Appendix

Driver taxonomy

Comparing praise/complaint to overall performance

Agent Arena: Code enables us to compare how models rank on the praise/complaint signal with their net improvement score - a holistic metric for relative model performance. We find that Anthropic and OpenAI occupy the top seven places on both rankings, and across all models shown, the Spearman correlation is 0.96. While some agreement is expected, as praise/complaint contributes to the net improvement score, the close alignment is significant.

Task outcomes offer another view of how robust this signal is. Agent Mode users get asked within their sessions to explicitly confirm if the tasks the agents are working on are being completed as planned. Our data shows that complaints were twice as common in tasks users judged unsuccessful as in those judged successful - 50.9% versus 25.9%.