Where agent time actually goes, and why it is rarely inference

Performance work follows attention, and attention goes to the model. It is the conspicuous component, it has a price per token attached, and it is the thing everyone reads about.

When you instrument an agent run properly, the picture is different enough to change what you work on.

The rough shape

Across the runs we have looked at, elapsed time splits roughly into four parts.

Model inference, which is real but usually a minority.

Retrieval, including whatever the index does and whatever the reranking stage does if one exists.

Tool calls, which reach outside your system and are therefore subject to somebody else's latency.

Sequential structure, which is the time lost to steps that ran one after another and did not need to.

The proportions vary by workload. The surprise is consistent: the last two are usually larger than teams expect, and they are the ones nobody is optimising.

Sequential structure is the cheapest win

An agent that retrieves, then checks a calendar, then looks up a record, then calls the model has paid four latencies in series. Frequently the first three are independent and could have run at once.

This looks like correct code rather than a performance problem, which is why it survives review. It is also usually the largest single improvement available and requires no change to models, prompts or infrastructure.

Tool calls are somebody else's problem, which makes them yours

A tool that reaches an internal system inherits that system's behaviour, including its bad afternoons. An agent with no timeout on a tool call has an unbounded worst case that will eventually occur.

Two things help. Timeouts with a defined fallback, so a slow dependency degrades the run rather than hanging it. And measuring tool latency separately per tool, so the one that occasionally takes eleven seconds is visible rather than averaged away.

Retries are time as well as money

We write about retries in cost terms often enough. They are also the single largest latency multiplier in most systems, because a failed run that restarts pays the full elapsed time twice for one outcome.

Any latency figure that averages successes with retries is describing something that does not happen to a user.

Measure percentiles, not means

Agent run durations are heavily skewed. A small proportion of runs take far longer than the rest, usually because of a slow tool, a retry, or an agent wandering through more steps than expected.

The mean of that distribution is a number that describes no actual run. The high percentiles are what a user experiences on a bad day, and they are what a latency budget should be set against.

Where to start

Instrument the four categories separately. Look at percentiles. Find the steps that are sequential and independent. Put timeouts on tools. Count retries apart from successes.

That work is unglamorous and typically yields more than a model change would, which is the recurring theme of nearly everything we write.

Next

The AI Observability pages cover instrumentation inside a run. The Agent Orchestration System pages cover evaluation, where latency is scored alongside cost and correctness rather than watched separately.