Announcing EnterpriseOps-Gym-AA, our independent leaderboard for ServiceNow's EnterpriseOps-Gym, testing whether AI agents can run real enterprise operations while following business rules and policy

We've partnered with @ServiceNow to launch EnterpriseOps-Gym-AA on Artificial Analysis, adapting the original EnterpriseOps-Gym benchmark (https://github.com/ServiceNow/EnterpriseOps-Gym) to run on our Stirrup agent harness. The benchmark drops agents into live enterprise systems and asks them to complete multi-step, stateful work across 8 business domains, with models graded on the final state of the underlying databases rather than the steps taken.

As we launch the eval, @AnthropicAI's Claude Fable 5 (max) leads at 51%, ahead of @GoogleDeepMind's Gemini 3.5 Flash (high, 50%), @OpenAI's GPT-5.5 (xhigh, 47%), with @Zai_org's GLM-5.2 (max, 43%) as the highest-scoring open weights model.

Key elements of EnterpriseOps-Gym-AA:

➤ Stateful, multi-step enterprise work: Agents operate live enterprise systems where actions are often irreversible. A wrong write can corrupt the environment the way it would in production, so agents cannot brute-force their way to a solution.

➤ Live tool use across 8 domains: Models act through live Model Context Protocol (MCP) servers over 164 database tables and 512 tools, spanning core business systems (Customer Service, HR, IT Service Management), collaboration tools (Email, Calendar, Teams, Drive), and cross-domain Hybrid tasks.

➤ Outcome-based SQL verification: Each task is graded by executable SQL checks on the final environment state, testing goal completion, state integrity, policy compliance, and unintended side effects. This rewards outcomes over rigid action sequences and allows alternative valid solution paths.

Key results:

➤ The frontier just clears 50%: Claude Fable 5 (max) leads at 51%, the only model clearly above half, with Gemini 3.5 Flash (high) close behind at 50% - a fast, low-cost Flash-tier model edging out larger frontier systems. Even the strongest models complete only about half of the 1,117 oracle-mode tasks.

➤ Models must pass all verification checks to succeed at a task: Models must satisfy all verification checks to succeed at a task, because a single missed constraint often means failure in the real world. We see that models satisfy most individual verifier checks (79% for the leader, Claude Fable 5) but complete far fewer whole tasks (51%).

➤ Open weights are within ~8 points: GLM-5.2 (max, 43%) is the strongest open weights model, about eight points behind the overall leader, with DeepSeek V4 Pro (max, 40%) and Kimi K2.7 (40%) close behind.

We would like to thank @ServiceNowRSRCH , @Mila_Quebec, @UMontreal and the benchmark authors for their excellent work building EnterpriseOps-Gym, and appreciate their collaboration in launching it on Artificial Analysis!