Insights · Approach
A pilot should be designed to end
Many AI pilots never finish. They drift on, neither adopted nor stopped, because nobody agreed in advance what success would look like or when the question would be settled.
A pilot is an experiment. Its purpose is to answer a question cheaply before the organization commits to the expensive version. That sounds obvious, yet a great many pilots are set up as if they were the first phase of a project that has already been approved. They have a start date and no end date. They are measured by whether the technology works, not by whether the business is better off.
The result is familiar: a tool that a few enthusiasts use, a cost that never quite goes away, and a leadership team that cannot say whether the investment was worth it.
Write the ending before you start
Before any work begins, we agree five things with the client, in writing.
- The question. One sentence. For example: can drafting first replies to routine enquiries with AI reduce the time our team spends on them without lowering quality?
- The measure. How we will know. Time per enquiry, error rate, the share of drafts sent with light edits, the view of the people doing the work. Measured before the pilot as well as during it, so there is something to compare against.
- The threshold. The result that would justify going further. If the measure does not reach it, the honest answer is no.
- The end date. A pilot that runs for a fixed number of weeks forces a decision. A pilot that runs until further notice avoids one.
- What happens next in each case. If it succeeds, what would a full rollout involve and cost? If it partly succeeds, what would we change? If it fails, how do we switch it off cleanly?
Use real work and real people
A pilot run on sample data by a project team proves very little. The people who will use the system should use it, on their actual work, for the whole period. Their judgment of whether it helps is one of the measures, and often the most important one.
That also means the pilot has to be safe to run on real work. Where personal or confidential information is involved, the handling of that information is settled before the pilot starts, not after.
Stopping is a result
An organization that runs a pilot, finds that the idea does not pay, and stops, has spent a small amount of money to avoid spending a large one. That is a good outcome, and it should be reported as one.
A pilot that ends with a clear no is a success. A pilot that never ends is the expensive failure.
Count the cost of keeping it
The last question is the one most often missed: if the pilot succeeds, what does it cost to run the system for real? Licences, usage charges, support, training new staff, keeping it current as the underlying services change. A pilot that saves time but costs more to run than the time is worth has answered its question, and the answer is no.
When a pilot is designed to end, the people involved know what they are testing, the leadership team knows what it is deciding, and the organization learns something whichever way the result falls.
NLM Cognitive Solutions
To discuss this article or your own situation, write to hello@nlmcognitive.com.