How to Tell If Your AI Agent Is Actually Working
How to Tell If Your AI Agent Is Actually Working
Two months in, someone is going to ask whether the agent is worth it. The honest answer requires knowing what to measure, and most of the numbers that are easiest to collect are the ones least connected to whether anything improved.
An agent produces a lot of visible motion. Messages, tool calls, tasks touched, hours of uptime. All of it is trivially countable, which is exactly why it ends up on dashboards. None of it distinguishes an agent doing useful work from an agent doing expensive nothing.
The vanity metrics
Messages sent. A chatty agent and a productive one look identical here. If anything the correlation runs backwards, because an agent that asks for confirmation constantly generates more traffic while delivering less.
Tokens burned or tasks touched. This measures consumption, not production. An agent that retries the same failing operation forty times posts impressive numbers.
Hours saved. Almost always a back-calculation from an assumption, not a measurement. Someone estimates a task used to take thirty minutes, multiplies by how often the agent ran, and produces a figure with no observation anywhere in it. If nobody actually got thirty minutes back, the number is fiction.
Uptime. Worth monitoring, useless as a value signal. A system available around the clock to do nothing useful is available around the clock.
The common failure is that each of these measures the agent's behaviour rather than the change in the work. The question is not how busy the agent was. It is whether outcomes moved.
The signals that matter
Tasks completed without rework. The single most informative number. Not tasks attempted, not tasks closed, but tasks that were finished and then never had to be touched again. Rework is where the cost of a mediocre agent hides: the work looks done, and someone quietly redoes it later. Track the rework rate and it stops being invisible.
Escalation rate, and its trend. How often the agent stops and asks. The absolute number matters less than the direction. Escalations falling over time means it is genuinely learning the boundaries of its job. Rising means something drifted. Flat at zero is not a triumph, it usually means the gates are too loose and you should check what went out unreviewed.
Time to first useful action. Wall-clock from a request arriving to something real happening. This is the metric users actually feel. An agent that produces a perfect result in four hours loses to one that produces a good result in four minutes, for most operational work.
Things that stopped falling through. The clearest evidence is often a category of failure that simply disappeared. Reports that used to be late. Follow-ups nobody sent. Checks that got skipped during busy weeks. Count those before and after, because this is the real product.
Human corrections per output. How much editing a result needs before it can be used. An agent whose output ships unmodified is worth many times one that needs a rewrite each time, even if their throughput numbers match.
What belongs on the dashboard
Keep it to five or six numbers, each of which would change a decision.
Completion rate without rework, weekly. Escalation rate with its trend line. Median time to first useful action. Count of failures the agent caught that a human would likely have missed. Count of incidents the agent caused, which should be tracked honestly and openly, because an agent with zero recorded incidents is usually under-instrumented rather than perfect.
Pair every efficiency number with a quality number. Throughput without a rework rate next to it is the metric most likely to be optimized into something harmful, since the fastest way to close more tasks is to do each one worse.
Measure the delta, not the agent
The most useful framing is comparative. Not "how did the agent do," but "what is different now."
That means writing down the baseline before deployment, which is the step almost everyone skips and then cannot recover. How long did this take? How often was it missed? How many times a month did someone have to chase it? Those numbers are unremarkable to collect beforehand and impossible to reconstruct afterwards, so collecting them is the whole trick.
Without a baseline, every conversation about value becomes a debate about impressions, and impressions favour whoever is most enthusiastic rather than whoever is right.
The test that settles it
If you are still unsure, try the cleanest experiment available: turn it off for a week and see what breaks.
If nothing does, you have your answer, and it is cheaper to learn that in a controlled week than after a year of renewals. If several things break immediately, the list of what broke is the value, described more precisely than any dashboard was managing. Either result is worth more than another month of debating the numbers.
Picking the handful of signals worth watching, and the baseline discipline that makes them mean anything, is covered in One Agent, One Company ($9.97). Useful if you are setting up measurement for an agent before you have to defend it to someone.