Why isn’t a single prompt-latency number enough to describe how an agentic coding workload performs?
A coding agent does not answer one isolated prompt and stop. It reads a task, calls tools, receives their output, changes the repository, and continues from there. MLCommons’ agentic inference work treats this as a trajectory of dependent turns rather than a single request-response pair: context grows across turns, output lengths vary, and the system has to preserve useful history under KV-cache pressure while it keeps making progress.
This article reviews public MLCommons documentation checked on 22 September 2026. It does not report benchmark numbers run by this site.
A coding agent answers more than one prompt
Treat a coding-agent task the way MLCommons treats it: as a trajectory of dependent turns. Time to first token still matters for a single response, but it doesn’t tell you whether the tenth turn of a long task is still fast, whether the agent’s growing context is being reused efficiently, or whether it eventually reaches a correct stopping point at all. A system that looks fast on the first turn can feel slow in practice if every later turn rebuilds context it should have kept. A high raw token rate can hide a genuinely poor tool-calling loop underneath it.
What should actually be measured
That reframes what a useful benchmark reports. End-to-end task duration and per-turn latency both matter, separately. So does context growth over the course of a trajectory, and whether cached context is actually being reused rather than recomputed. MLPerf’s v6.1 results extend this thinking to edge-agentic and end-to-end RAG workloads specifically, which is a sign the field is moving past a single latency number as the headline metric for anything agentic. Whether the agent reaches a correct stopping point is its own separate measurement, and needs a definition of “correct” decided before you look at a single result.
Build the benchmark around the product
The benchmark should match the product. Replay a fixed, representative set of tasks. Record the exact model and serving configuration used: results still describe that configuration, and they cannot be assumed to describe a changed one. Decide the definition of correctness before you look at the output, to avoid unconsciously grading toward whatever the system happened to produce. Keep accuracy and throughput as separate scorecards; a fast wrong answer and a slow correct one fail differently, and averaging them together hides which one you actually got.
The shift is methodological
Do not turn a vendor’s single-turn benchmark number into an implicit promise about how their system performs on real repository work; the two are measuring different things. The interesting shift here is methodological: as coding agents become longer-lived, performance engineering has to measure how a task progresses across turns, turn by turn, rather than only the output of one request.
Follow the English RSS feed for the next article.
About the author: see my portfolio and my agency, Bimbi Digital.