Meet Jeff. Jeff's real name isn't Jeff, but for the sake of his anonymity we'll call him that. He runs operations at an energy consultancy, and he built an AI tool to take over his team's chasing of overdue client data. It worked through the team's tracking spreadsheet, wrote the emails and sent them. The pilot looked good. Then a client replied: "This is the third reminder. We're still waiting for you to tell us which sites to include." The spreadsheet said "overdue," which was accurate. But his team would normally read the correspondence first, and chase their own consultant instead. That step had never been written down. Everyone doing the job already knew to do it.
Nothing in the spreadsheet told the tool to check whether the hold-up was on the consultancy's side, so it kept sending reminders. Many organizations are in a similar position when they try to replace a process with AI. The process they describe is the documented one, which leaves out the checks people make without thinking about them. Few of these processes are measured, so there is no baseline to compare the AI against either, and a pilot can run for weeks before anyone can say whether it is better or worse than what it replaced.
Exhibit 1.1
The written process left out the check that mattered
The process as written
The process as the team ran it
Read the latest correspondence with the client
Send the client a reminder
Chase our own consultant
How well does the current process work?
Teams often want to discuss models and vendors before they've measured how well their existing process works. Without that baseline, it's hard to know whether AI would improve anything.
That usually isn't carelessness. Most of these processes have never needed formal measurement. They worked well enough, the people running them knew what a good result looked like, and problems got fixed by whoever noticed them.
Once you want to replace a process, you need to know more about it. Before considering AI, I would want to know how the output is checked today, how success is measured, and what happens when something goes wrong. Most organizations can't answer all three for the processes they want to automate.
Three things to know before you replace a process
1. How is the output checked?
Find out who checks the work today, what they look at, and how often. In many processes nobody has that job formally, or it falls to a team lead when they have time.
To check whether a quality standard exists, pull 20 recent outputs and ask two experienced people to mark each one as good or not good, separately. If they often disagree, the standard lives in individual judgment. You would have nothing consistent to evaluate an AI against, and a vendor demo would be judged against examples the vendor chose.
2. How is success measured?
You need current figures for things like error rate, turnaround time and cost per item. If nobody has them, measure the process for a few weeks before a pilot starts, then agree on the result the AI version has to beat. If that number is set after the pilot, it is too easy to adjust it to whatever the pilot achieved.
In a 2025 randomized trial by METR, experienced open-source developers took 19% longer to finish tasks when they used AI tools. Afterwards they estimated the AI had made them about 20% faster. Ask your team how a new tool is working, but check their answer against the numbers.
Exhibit 1.2
Developers estimated that AI cut their time per task by 20%, when it had increased it by 19%
Developers forecast that AI would cut their time per task by 24%, and afterwards estimated that it had cut it by 20%. Tasks done with AI took 19% longer.
Source: METR, randomized controlled trial with 16 experienced open-source developers working on 246 tasks in their own projects, early 2025.
3. What happens when it fails?
When an output is wrong, it matters who notices, how long that takes, who fixes it, what it costs and who is accountable. In many processes these answers are informal and spread across several people.
Take the last three times the process went wrong and trace each one from the mistake to the person who caught it and the fix. If you can't find three recent failures, either the process is very reliable or its errors aren't being noticed. It's worth finding out which.
In 2022, Air Canada's website chatbot told a customer whose grandmother had died that they could buy a full-fare ticket and apply for a bereavement refund within 90 days. The airline's policy didn't allow that. When the customer took the case to a tribunal in British Columbia, Air Canada argued that it could not be held liable for information provided by its agents or representatives, including a chatbot. The tribunal rejected that argument in 2024. It treated the chatbot's answers like any other page on the airline's website, so the liability sat with Air Canada.
Checks that nobody wrote down
When a process isn't defined, the people running it do much of the quality control themselves. Jeff's team read the correspondence before chasing anyone. An experienced claims handler knows which customers tend to send the wrong form, and a finance clerk will query an invoice that is technically correct but doesn't look right. These checks rarely appear in the process documentation, if there is any.
If an AI tool replaces those people, the checks go too. Because they were never measured, the loss is hard to see until something goes wrong.
In early 2024, Klarna said its AI assistant was doing the work of 700 full-time customer service agents, with customer satisfaction on a par with human agents. In 2025 it started hiring human agents again. Its CEO, Sebastian Siemiatkowski, said cost had been "a too predominant evaluation factor" and that the result was lower quality. The figures published at launch looked good, and the quality problems the CEO described came later.
In 2025, Deloitte Australia agreed to partially refund the Australian government for a A$440,000 report that contained a fabricated quote from a federal court judgment and references to research papers that don't exist. A revised version disclosed that generative AI had been used to write it. Whatever review the report went through before publication did not catch those errors. Checking that a cited source exists is a basic step, and it matters more when part of a document is generated by a model.
Google's 2024 DORA report, which surveys software teams, found that a 25% increase in AI adoption was associated with a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. The authors' explanation is that AI makes it easy to write more code at once, so changes get bigger, and DORA's research has long shown that bigger changes carry more risk. The tool's effect depended on practices around it, such as how changes are sized, reviewed and tested.
In each of these cases the AI was working inside a process whose checks, measures or ownership were incomplete, and the gaps showed up after it went live.
If you can't answer the three questions
I would hold the AI pilot and start a smaller piece of work first. Define what a good output looks like, measure the process as it runs today, and map what happens when it fails. The people who run the process can do most of this, and they know the steps that aren't written down.
Once that's done, you are likely to be in one of three positions.
- The process works well enough that replacing it is hard to justify.
- The process has problems that AI won't fix, such as missing information, unclear ownership or a step nobody needs. Fixing those may be enough.
- You understand the process well enough to automate parts of it, and you have a baseline to measure the AI against and a way to catch its failures.
Whether AI fits
Knowing how a process works is the starting point for deciding whether AI suits it. That decision depends on other questions, such as whether checking the AI's output costs less than doing the work, whether this step is actually the bottleneck, and whether something simpler would do the job. I'll cover those in the next post.
Checklist before an AI pilot
Work through this with the people who run the process before you replace it with AI.
Download the process baseline worksheet (PDF, one page) to fill in with the people who run the process.
How the output is checked
- Name the person who checks the output today, what they look at and how often they do it.
- Pull 20 recent outputs and ask two experienced people to mark each one as good or not good, separately. Note how often they disagree.
- Write down what a good output looks like, in terms both of them accept.
- Ask the people who run the process which checks they make that aren't in the documentation, such as reading the latest correspondence before chasing a client.
How success is measured
- Record today's error rate, turnaround time and cost per item. If nobody has these figures, measure the process for a few weeks.
- Agree on the result the AI version has to beat, and write it down before the pilot starts.
- During the pilot, check the team's view of the tool against the figures you recorded.
What happens when it fails
- Trace the last three failures from the mistake to the fix. Note who noticed, how long that took, who fixed it, what it cost and who was accountable.
- If you can't find three recent failures, find out whether the process is reliable or its errors are going unnoticed.
- Decide who will notice and fix the AI's mistakes once it is live, and who is accountable for them.
If any item is still open, finish it before you start the pilot. In the next part, we take a look at the four questions you need to answer before you consider adding an AI feature to your product.