Our co-founders Micah (@_micah_h) and George (@grmcameron) spoke at AI Engineer World’s Fair about the cost of intelligence for agentic tasks.

While accessing a fixed level of intelligence is becoming cheaper, the cost of completing complex agentic tasks is increasing. As models take on more complex agentic real-world tasks, they have larger context requirements and are relying on additional turns and tool calls which lead to higher cost per task.

To reflect these workloads and capabilities, we developed AA-Briefcase, our new agentic knowledge work benchmark, designed to measure models on long-horizon knowledge work tasks. Models are evaluated on tasks across multi-week projects and several thousand input files.

As discussed in the talk, cost per task is becoming an increasingly key metric and is a combination of the four following factors:

➤ Token price: The price of accessing a given level of intelligence continues to decline on a per-token basis. However, as models become capable of completing longer horizon work, intelligence and token usage have grown faster, resulting in a higher overall cost per task.

➤ Turns: Agentic work requires multiple ‘turns.’ All turns are included in the context of subsequent turns, converting prior output tokens into input tokens and causing costs to accumulate throughout the trajectory, even with input caching discounts. Some models require significantly more turns than others, without necessarily higher performance.

➤ Token efficiency: Models vary significantly in how many tokens they use to complete the same task. A more verbose model may cost more per task even when its token prices are lower. Looking at the broader picture of ‘cost per task’ helps to contextualize token efficiency and token cost.

➤ Prompt caching & hit rate: Long agentic tasks repeatedly pass the same instructions, research, tool outputs, and prior work back into the model. Prompt caching reduces the price of these reused input tokens, making the difference between standard input pricing and cache-hit pricing increasingly important as the context increases.

Watch the full keynote: https://www.youtube.com/live/4sX_He5c4sI?si=H_PFO8mDE75cn9LP&t=29106