Choosing a model is an operations decision, not a benchmark decision

There is a familiar sequence. A team compares models, picks one on published scores, builds around it, and finds that the agent behaves unpredictably in ways the comparison never surfaced. So they try another model. The behaviour changes; the unpredictability does not.

The diagnosis is usually the same: the model was never the variable that mattered.

What benchmarks can and cannot tell you

A published score tells you how a model performed on a fixed set of problems under conditions somebody else chose. That is genuinely useful information about capability in general, and nearly useless as a prediction about your specific work.

Your work has context they did not have. Your documents, your conventions, your idea of an acceptable answer, your tolerance for a confident wrong response versus a hedged right one. A model that scores marginally higher in the abstract can easily be the worse choice for a workflow where failure has a particular shape.

This is not an argument against benchmarks. It is an argument against treating them as a substitute for evaluating against your own work.

Build the evaluation before choosing

The practical advice is unglamorous: assemble thirty to fifty real examples from the work you actually intend to automate, with the output you would consider correct. That set is worth more than any comparison table, for three reasons.

It measures the thing you care about. It keeps measuring it when you change something else. And it makes model choice reversible, because swapping a model becomes an experiment with a result rather than a migration with a hope.

Teams that build this set early tend to make model decisions quickly and stop revisiting them. Teams that skip it revisit the decision indefinitely, because nothing ever settles it.

Why the same model behaves differently for two teams

Almost everything around the model is a variable. The retrieval strategy determines what the model sees. The tool set determines what it can attempt and how much context it carries before it starts. The prompt accumulates edge-case handling over months. The failure policy determines whether a shaky answer is returned or held.

Change any of those and behaviour changes measurably, usually by more than a model swap would. That is why we treat model choice as one setting inside an operational system rather than the foundation of it.

Where cost enters

Model pricing is the most visible cost and rarely the largest. Retries, oversized context and unused tool definitions routinely account for more, and none of them change when you switch model.

This is why the Optimizer measures model choice, prompt shape and tool set together, against your evals. Moving to a cheaper model is only a saving if quality holds on your own criteria. Sometimes it does, and the saving is real. Sometimes it does not, and you have traded a visible cost for an invisible one.

Our actual recommendation

Pick a capable model, build your evaluation set, instrument cost per agent, and then spend your attention on retrieval, tools, policy and review. Revisit the model when your evals tell you to, not when a new score is published.

That is a less exciting answer than a comparison table. It is the one that has produced stable systems in the deployments we have watched.

Next

The versions page covers every generation of the Yaju Agent Orchestration System and what each one added. The Agent Orchestration System pages cover the evaluation and optimisation layers that make model choice a reversible decision.