To tell whether an AI tool is really saving your team time, pick one task, time it properly before the tool arrives, then time it again with the tool, counting the checking and rework as well as the drafting. Track the mistakes that get through, add up what the tool costs, and decide in advance the date and the rule for keeping it or dropping it. Anything less is a feeling, not a measurement.
Why "it feels faster" isn't enough
AI tools are very good at the part of a job that is easy to notice. A draft appears in seconds, so the work feels finished. The parts that are easy to miss come afterwards: reading the draft properly, fixing what it got wrong, chasing the detail it couldn't know, and cleaning up the occasional mistake that reached a customer. If you only notice the fast part, almost any tool looks like a win.
There is also a pull towards believing it. You chose the tool, you paid for it, and you told the team it would help. That is exactly when a simple, written measurement earns its keep. It protects you from keeping a tool that doesn't help, and it gives you real evidence when one does.
Is AI actually saving your team time? Start with one task
Don't try to measure "AI" across the whole business. Pick one task that happens often, has a clear start and finish, and is done by the same people each time. Good candidates are things like first replies to routine customer questions, product descriptions for new lines, or summarising a long supplier thread before a call.
Write down what "done" means for that task before you measure anything. A customer reply is done when it is sent and correct, not when a draft exists. A product description is done when it is live and accurate, not when the text is generated. If the finish line moves between the before and after measurements, the comparison is meaningless.
Time it before you start
The baseline is the step most people skip, and without it there is nothing to compare against. For a week or two before the tool arrives, record how long the task takes from start to finish. Keep it simple:
- the date and who did it;
- when they started and when the task was done, by your definition of done;
- anything unusual, such as an especially complicated case.
A shared spreadsheet is enough. Record enough instances to see the normal range, not just one good day. If the task varies a lot, note a rough size for each one (short, medium, long) so you compare like with like later.
Count the checking and rework, not just the drafting
Once the tool is in use, time the whole task again, using the same definition of done. This time, split it into parts:
- Preparing: gathering what the tool needs and writing the request.
- Generating: waiting for the output.
- Checking: reading the output against the source and spotting problems.
- Fixing: rewriting, correcting facts, or starting again by hand.
The checking and fixing rows are where the honest answer usually lives. A tool that drafts in seconds but needs careful line-by-line checking may save far less than it seems to, or nothing at all. That isn't a reason to skip the checking: for anything a customer sees, it is the job. It is a reason to count it.
Also record the times the tool wasn't used, or was abandoned halfway. If people quietly go back to doing the task by hand for the hard cases, the average with the tool only covers the easy ones.
Track errors as well as speed
A faster process that lets more mistakes through is not a saving; it moves the cost somewhere else, often to a customer or to someone further down the line. So alongside the time, keep a short count of problems for both periods, before and after:
- mistakes caught during checking;
- mistakes that got through, such as a wrong detail in a reply or a spec that doesn't match the product;
- follow-ups caused by the work, such as a customer writing back because the first answer missed their question.
Mistakes that get through deserve extra weight, because each one costs time to put right and some cost trust. If the error count rises while the time falls, look closely before you call it a win. Our post on AI product descriptions that don't make things up covers one way to keep errors down in catalogue work.
How to measure the return on an AI tool
You don't need a complex model to work out the return on an AI tool. For a small team, a plain comparison is enough. Work it out per month:
- Time saved: the average time per task before, minus the average time per task after (including checking and fixing), multiplied by how many times the task happens.
- Minus the time it costs: setting the tool up, writing and maintaining instructions, training people, and cleaning up mistakes that got through.
- Minus the money it costs: subscriptions, usage charges, and any paid help to set it up.
What a saved hour is worth to you is your call. It might be wages, or it might be what that person can now do instead. Write down which you are using, so the next review uses the same rule.
Be honest about whether the saved time is real. Ten minutes saved on each of many tasks only matters if the time goes somewhere useful. If it disappears into the day, the tool hasn't made a difference to the business, however good the numbers look.
Set a review date and a keep-or-drop rule in advance
Decide two things before the trial starts, and write them down:
- When you will review it. Pick a date far enough away for people to get past the learning stage, but close enough that a poor tool doesn't become a habit.
- What result keeps it. For example: "We keep it if the task takes clearly less time, including checking, and mistakes that get through are no higher than before." Use your own wording, but set it before you see the results.
Writing the rule first matters because it is very easy to move the goalposts once you have the numbers in front of you. If the result is unclear, that is an answer too: run a narrower trial, change how the tool is used, or drop it.
When a tool passes, the same thinking helps you decide how much it should do on its own. Our guide to deciding what an AI agent may do: act, draft or escalate covers that next step.
Be wary of vendor claims, ours included
Many AI tools come with headline figures about time saved. Treat them as a reason to run your own test, not as a result. A vendor's figure was measured on someone else's task, with someone else's definition of done, and often without counting the checking. It may be true for them and still not true for you.
That applies to anything we sell too. If one of our tools doesn't save your team time on your work, measured the way this post describes, it isn't worth paying for.
A simple measurement sheet
If you want somewhere to start, one sheet with these columns covers most of it:
- date, person and task size;
- with or without the tool;
- minutes preparing, generating, checking and fixing (or total minutes for the baseline);
- mistakes caught, mistakes that got through, and follow-ups;
- a short note on anything unusual.
Keep the totals and averages in formulas rather than working them out by hand, so the review date is a matter of reading the sheet. For more on choosing which tools are worth trying in the first place, see the AI stack to get things done fast.
Measure one task honestly before you roll a tool out further. The answer is often "yes, but only for part of the job", and that is still a useful thing to know.