Many owners now run AI agents. Ask what the agents changed, and you may hear what they did instead. An agent's own report usually counts what it did: chats handled, emails sent, hours saved. That is activity, and agents are built to produce it. The short answer: an agent is useful only if a number your business tracked before it existed moves, after you subtract the time your people spend checking and fixing its work. By the end of this article you'll have four rules to judge any agent, three questions that decide on a set date whether it grows or goes, and a one-week plan to put your first agent on trial.
Four rules decide whether an agent is useful, and each fits on one line
Use them in order, one agent at a time. Each rule is a test you can run yourself.
Rule 1: Give each agent one number that existed before it did
Do this. For each agent, write one line: "This agent exists to move [number] from [last four weeks] to [target]." Good numbers: hours a person spends on the job each week, days to send an invoice, leads answered within an hour, cost per order. Bad numbers: anything the agent reports about itself.
It worked when you can say before and after in one sentence, with no slide.
Why. An agent's own counts start on the day it starts, so they can only go up. A number with a past shows you a change.
Example. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied an AI assistant used by 5,172 customer support agents. They didn't count the assistant's replies. They counted issues resolved per hour, before and after it arrived. That rose 15% on average and about 30% for new and less-skilled staff, with little gain for the most experienced 1. One old number, and who it helped.
Rule 2: Subtract the minutes your people spend checking and fixing its work
Do this. For two weeks, everyone who touches the agent's output notes the minutes they spend on it. Add the monthly bill, including usage fees. Take both away from what the agent saved.
It worked when you have a net number, and it is still above zero.
Why. A common question is what an agent costs to keep running. The bill is the easy part. The minutes your team spends reading, checking and fixing never show on a dashboard.
Example. Workday surveyed 3,200 people in November 2025. Of them, 85% said AI saved them one to seven hours a week. But nearly 40% of that time went back into rework. Only 14% got clear, positive results consistently 2. BetterUp Labs and the Stanford Social Media Lab call the problem "workslop": AI work "that looks polished and complete" but is "unhelpful, low-quality, or off the mark." Of the US desk workers they surveyed, 40% believed they had received some in the past month, and each case took nearly two hours to deal with 3. A Danish study of 25,000 workers found that AI chatbots had created new tasks for 8.4% of them, including some who don't use the tools 4.
Rule 3: Test it against the old way, because you can't feel the difference
Do this. For two to four weeks, split the work. Every other job goes to the agent, and the rest runs the old way. Compare the number from Rule 1, after the minutes from Rule 2.
It worked when you have two numbers side by side, from the same weeks.
Why. People often judge tools by how they feel to use, and that feeling can point the wrong way.
Example. In 2025 the research group METR ran a trial with 16 experienced software developers on 246 real tasks. With AI allowed, they took 19% longer. Before the trial, they expected AI to speed them up by 24%. Afterwards, they still believed it had sped them up by 20% 5. METR now says these results are out of date. The lesson about feelings stands: these experts felt faster while they were slower.
Rule 4: Write the switch-off line before the trial, and count the mistakes customers see
Do this. Write one sentence before you start: "If [number] hasn't moved from X to Y, after fixing time, by [date], the agent goes off." Add one customer rule: one wrong answer to a customer about money, contracts or safety sends the agent back to drafts that a person approves.
It worked when the decision on the date takes no meeting.
Why. Once an agent is running, switching it off can feel like a loss, so the decision can drift. A line written in advance ends the debate. And an agent's mistakes are your mistakes.
Example. Air Canada's website chatbot told a customer he could claim a bereavement refund after travel, within 90 days of the ticket being issued. The policy page it linked to said the refund didn't apply after travel. Air Canada argued it couldn't be held responsible for what the chatbot said. In February 2024 a Canadian tribunal disagreed. "It makes no difference whether the information comes from a static page or a chatbot," the tribunal member wrote. The airline had to pay $812.02 6. The sum was small. The rule is the point.
On the date you wrote, three questions decide.
An agent's own report usually looks good, because it counts tasks, not results
Some put it bluntly: their agents are "a high output team that delivers absolute no value whatsoever." Others are blunter still: "most of it is junk, and a waste of tokens." Yet the hope behind the agents is real. One owner put it this way: they are setting up agents "so I basically can scale 'labor' before my business can afford human labor."
The gap between the hope and the result has a simple cause. A chatbot answers when you ask. An agent is handed a job and works through the steps on its own, across your tools. That makes it the purest task worker there is. Its report usually counts tasks: chats handled, emails sent, records updated. Judge any worker by tasks done and a busy one always passes.
But you didn't buy the agent for tasks. You bought it to cut a cost, win a sale or free a person. That is a number, and it isn't on the agent's report.
The usual advice teaches you how to build an agent and how to onboard it like a new hire. It is good advice, and it usually stops before the next question: is this one worth keeping? Most projects never show a payoff. A 2025 MIT NANDA report reviewed more than 300 public AI deployments. About 5% of generative AI pilots reached fast revenue growth. The rest stalled, with little to no measurable effect on profit 7. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, because of rising costs, unclear business value or weak risk controls 8.
You buy software and count what it does: logins, tickets, reports sent. The tool's own dashboard says it works.
You give each AI agent one number your business already tracked, and count what changed after your team's fixing time. The agent's own counts don't decide.
This is the move from level 4, where AI does measured work, to level 5, where AI carries real work and you grow without matching headcount. The four rules are how you make it.
The best-known AI service stories counted work, and the useful result counted a number
Klarna is a well-known example. In February 2024 it said its AI assistant had handled 2.3 million chats in its first month. That was two-thirds of its service chats and the work of 700 full-time agents. Klarna said customers were as satisfied as with human staff, and estimated a $40 million profit improvement for 2024 9. The first numbers count work. The last one was a forecast. In May 2025 its chief executive, Sebastian Siemiatkowski, told Bloomberg that Klarna, still using the AI, was hiring human staff again. "Really, investing in the quality of human support is the way of the future for us," he said 10.
In July 2025 Commonwealth Bank of Australia announced that 45 customer service roles would be replaced by an AI voice bot. The workers' union said call volumes were rising, staff were being offered overtime and team leaders were answering some calls. In August the bank reversed the cuts and apologised. It said its first assessment "did not adequately consider all relevant business considerations," and that this "error meant the roles were not redundant" 11.
Now set the support-desk study from Rule 1 beside them. It didn't count the assistant's replies. It counted an outcome that existed before the assistant, issues resolved per hour, and it showed who gained and who didn't 1.
| case | what was counted | what the count hid | what happened |
|---|---|---|---|
| Klarna, 2024 to 2025 | Chats handled, the work of 700 agents, a $40M profit forecast | How service quality held up over time | Human staff hired again for quality (May 2025); AI kept |
| Commonwealth Bank, 2025 | A fall in calls, credited to the bot | Call volumes were rising, per the union | 45 cuts reversed, with an apology (Aug 2025) |
| Support-desk study, 5,172 agents | Issues resolved per hour, before and after | A small quality drop for top agents | Up 15% on average, about 30% for newer, less-skilled staff |
Judge an agent too early and you compare a trainee with a veteran
These rules can go wrong in four ways.
Switching off too soon. A process your team has run for years has usually been tuned along the way. A new agent hasn't. Give it the same effort your people got before you judge it: the written steps and the training. Start the trial after that, not on the day you switch it on. Freedom grows with a record: drafts first, then more.
Small volumes. If a job happens five times a week, four weeks give you twenty cases. That is too few to tell a small change from luck. For these jobs, use only Rules 2 and 4: does it save more minutes than it costs, and does it keep mistakes away from customers?
Payoffs that aren't money yet. An agent that answers leads at 2 a.m. may not show up in revenue for months. It still has a number: the share of leads answered within an hour, or how many become appointments. If you truly can't name one, call it a learning project. Give it a fixed budget and an end date, and don't expand it until it has a number.
The fear of falling behind. A fear many owners share fits in one line: "I will not be left behind!" The fear is fair. But the cases above show the other risk: moving fast, counting activity, and then reversing course in public.
This week, put one agent on trial with a number and a date
Put one agent on trial
On the date, ask the three questions. Grow it or switch it off, then move to the next agent.
You can now tell a busy agent from a useful one
You started with agents that report what they did and a business that can't say what changed. You now have:
- Four rules: one number that existed before the agent, net of your team's fixing time, tested against the old way, with a switch-off line written first.
- Three questions for the date: did the number move, is it still ahead after fixing time and the bill, and did mistakes stay away from customers?
- A one-week trial for the agent your team fixes most.
A busy agent shows you what it did. A useful one shows up in a number you already had.
Read before: an agent runs best on a process that already works by hand. Start with How to Build Systems in Your Business That Your Team Actually Uses, or see why nothing changed after you read the business books.
Read next: once agents carry real work, decide what you still do yourself in How to Replace Yourself in Your Business. If your agents help with posts, judge them by what the posts bring in, not how many go out: see Founder-Led Content.
Where you stand: the Operator 2030 Scorecard's Technology questions ask which workflows AI runs in your business, with "the baseline and the after number", and whether your tools "get killed on their numbers".
Sources
- Generative AI at Work · Erik Brynjolfsson, Danielle Li, Lindsey Raymond, The Quarterly Journal of Economics 140(2), 2025 (first as NBER Working Paper 31161, 2023)
- New Workday research: companies are leaving AI gains on the table · Workday, 2026-01-14
- The hidden costs of workslop · BetterUp Labs with the Stanford Social Media Lab, 2025-09-29
- Large Language Models, Small Labor Market Effects · Anders Humlum and Emilie Vestergaard, Becker Friedman Institute Working Paper 2025-56, 2025-05
- Measuring the impact of early-2025 AI on experienced open-source developer productivity · METR, 2025-07-10
- Air Canada has to pay for chatbot mistake · MobileSyrup, 2024-02-16
- MIT report: 95% of generative AI pilots at companies are failing · Fortune, 2025-08-18 (on MIT NANDA, "The GenAI Divide: State of AI in Business 2025")
- Over 40 percent of agentic AI projects could face the axe by end of 2027: Gartner · W.Media, 2025-06-27 (on Gartner's press release of 2025-06-25)
- Klarna unveils AI assistant powered by OpenAI; carries workload of 700 full-time agents · Silicon Canals, 2024-02-28
- Klarna is hiring customer service agents after AI couldn't cut it on calls · Entrepreneur, 2025-05-09
- CBA reverses AI-driven job cuts, admits 'error' · Information Age (ACS), 2025-08-21
