New Harvard + Stanford paper says, don’t replace your embedding model with an LLM. Use embeddings for the cheap, fast first pass.

And bring in an LLM only when the retrieval problem actually requires reasoning.

An LLM costs up to 1,431X more than an embedding model of comparable quality.

Across 10 LLMs and 26 embedding models on 37 tasks, Gemini 3.1 Pro scored 77.6 versus 77.2 for Octen-8B, a statistical tie.

The cost was nowhere close: $154.14 for the LLM benchmark pass versus $0.11 for Octen-8B, or 1,431× more.

The task split explains when that extra spend can make sense.

LLMs led retrieval by 8.5 points, while embeddings led classification by 5.6; clustering, semantic similarity, and pair classification were effectively tied.

Embeddings encode documents once and reuse vectors, while an LLM can read multiple documents together with the query and reason across them.

So overall recommendation, embeddings for candidate retrieval, LLMs only where the shortlist actually needs reasoning.

– arxiv. org/abs/2608.12875

Title: "The Embedder's Dilemma: LLMs Are Better, but at What Cost?"