Pilot Purgatory: A 90-Day Test to Get AI Into Production — or Kill It
MIT's bar was production with measurable impact inside about six months — and only about 5% of the custom tools it tracked cleared it. This page gives your pilot the same honesty on a 90-day clock: a baseline before, one finished task, and a kill-or-scale call you can defend.
What pilot purgatory is — and what it is not
Pilot purgatory is the state where an AI pilot is alive enough to keep paying for and too unmeasured to ever approve, kill or scale. It has no baseline, no named owner and no decision date, so every review meeting ends with “let's give it another month.”
A plain-language note on the phrase itself: “pilot purgatory” does not appear in the MIT report. It is industry slang coined by vendors and commentators after MIT NANDA's GenAI Divide report landed. We use it here because it is the phrase owners search for — never as a quote from MIT. What MIT did measure is the funnel underneath it: about 60% of organisations evaluated custom GenAI tools, about 20% piloted them, and about 5% reached production with marked and sustained impact. Most pilots do not die in a dramatic failure. They stall between the second and third numbers.
This page turns that finding into an AI proof of concept test you control. Ninety days is our operator test, derived from MIT's bar — MIT used about six months, and its own appendix warns that window may understate slower rollouts. A small firm does not need six months to learn whether one task got cheaper. It needs a before number and a date.
Why AI pilots stall: three fixable causes
- No baseline. Nobody timed the task before the tool arrived, so “it feels faster” is the only evidence on offer. Feelings do not survive a renewal invoice.
- No owner. The pilot belongs to everyone's enthusiasm and nobody's calendar. When it wobbles, nobody is charged with fixing the workflow or ending it.
- The tool does not learn the workflow. MIT calls this the learning gap: tools that do not retain feedback, remember context or fit day-to-day work get abandoned for mission-critical tasks, however good the demo was. Staff then quietly cross back to personal tools — the shadow AI gap — and the official pilot reports activity while the real work happens elsewhere.
Notice what is not on that list: the model. Model quality is rarely the binding constraint on a small firm's pilot. Measurement and workflow fit are.
The 90-day test, week by week
Run one pilot at a time. One task, one owner, one baseline, one decision date — written down before the tool is switched on.
Week 1: Pick one finished task
Choose a task with a countable finished output: invoices extracted, enquiries triaged, meeting notes issued, first drafts produced. Not “improve communication.” Score candidates first with the AI use-case scorer — time taken, repetitiveness, clean inputs, how easily errors are caught, and the cost of a mistake. The highest scorer is your pilot.
Weeks 1–2: Take the baseline before anything changes
Time the task for at least a week as it runs today: minutes per finished item, items per week, error or rework rate, and any external spend attached to it. Write the numbers where the day-90 meeting can see them. This is the step the stalled 95% skipped, and it cannot be reconstructed afterwards.
Weeks 2–3: Sanction the tool and name the owner
One approved tool, in an account the firm owns, with one named owner responsible for the count and the call. Set the data boundary at the same time — what may be entered, what must never be — using the AI governance checklist. Tell the team what is being tested and when it ends.
Weeks 4–8: Count finished work weekly
Every week the owner records the same three numbers as the baseline: finished items, minutes per item including human review, and errors caught. Fifteen minutes, same day each week. If the count slips for two weeks, that is data, not a reason to stop counting. Price the run so far in the AI ROI calculator at week 8, while there is still time to fix the workflow rather than the verdict.
Weeks 9–12: Widen carefully or hold the line
If the numbers hold, extend the task to the next person or the next document type — nothing broader. If they do not, use these weeks to test one redesign: cleaner inputs, a tighter template, human review moved earlier. One redesign only. A pilot that needs a fourth rescue is telling you its answer.
Day 90: Make the kill, scale or redesign call
The owner brings one page: baseline versus now, cost per finished task including review time and tool cost, error rate, and a recommendation. Scale if cost per finished task fell and quality held. Kill if it did not move or moved only by shifting work onto reviewers. Redesign once if the task is right but the inputs were the problem — with a new 30-day date, not another open ending. Whatever the call, write it into the procedure or switch the tool off. Both are successes compared with a second quarter of purgatory.
A worked example — illustrative, not a study
Example — copy the arithmetic, replace every number with your own
Invoice details into the accounting system
A small firm receives 60 supplier invoices a week. In the baseline week, a staff member spends 6 minutes per invoice keying details and fixing mismatches: 360 minutes, or 6 hours a week. Review and corrections are already inside that 6 minutes.
Under the pilot, the tool extracts the details and the same person checks each one: 2.5 minutes per invoice including review, or 150 minutes a week. Time saved: 210 minutes — 3.5 hours a week.
Value it with your loaded hourly cost. At an illustrative $40 an hour, 3.5 hours is $140 a week. If the tool costs, say, $60 a month in this example, the pilot pays on time alone — if the error rate in the weekly count did not rise. If errors rose enough that month-end cleanup ate an extra 2 hours a week, the honest saving falls to 1.5 hours and the day-90 call gets harder. That is the point of counting errors next to minutes.
These figures are invented for the example so you can see the shape of the arithmetic. They are not survey results, vendor claims or a promise about your invoices. Your baseline week replaces every one of them.
Run your own version on the AI ROI page, which prices cost per finished task, and check whether the firm around the pilot is ready with the AI readiness assessment.
The six-month caveat, stated plainly
MIT assessed ROI about six months after pilot and warned in its own appendix that this may be too short for complex, slow rollouts, potentially understating success. Fair. Our 90 days is not a claim that every worthwhile system pays in a quarter. It is a claim that a small firm can know, within 90 days, whether one narrow task is moving — minutes, finished count and errors — even if the full financial effect lands later. If your pilot genuinely needs longer, say so at week 1, set the longer date in writing, and still count weekly. Purgatory is not a long pilot. Purgatory is a pilot with no date at all.
Stuck on which task, or whether your baseline is honest? Talk it through with Ada — free, no signup
Tell Ada the task you are piloting and how long it takes today. You will get a straight answer on whether it can pass a 90-day test, what to measure first, and where AI should not touch it — on the way to a free report. Anything paid is discussed only after that report.
Frequently asked questions
What is pilot purgatory?
The state where an AI pilot is alive enough to keep paying for and too unmeasured to approve, kill or scale. It has no baseline, no named owner and no decision date. The phrase is industry slang; it does not appear in MIT's GenAI Divide report.
What is an AI proof of concept?
A time-boxed test of one AI task with a baseline taken beforehand, a named owner, weekly counts of finished work, and a decision date. A proof of concept that cannot be killed on its date is not a test; it is a subscription.
How long should an AI pilot run before we decide?
For one narrow task in a small firm, 90 days is enough to see whether minutes per finished task and error rates are moving. MIT used about six months and warned that may understate slower rollouts. Either way, set the date at the start and count weekly.
What should we measure in an AI pilot?
Finished items per week, minutes per finished item including human review, error or rework rate, and cost per finished task including the tool. Take all four before the tool starts, or the pilot cannot prove anything.
When should we kill an AI pilot?
On the decision date, if cost per finished task has not fallen, if quality only held by shifting work onto reviewers, or if staff have routed around the tool to personal accounts. Killing on evidence is a cheap success compared with another quarter of unmeasured spend.
Does the MIT report really say 95% of AI pilots fail?
No. It found about 95% of organisations in its sample saw no measurable P&L impact within about six months of a pilot. Of custom tools, about 60% were evaluated, 20% piloted and 5% reached production — roughly one in four pilots clearing a high bar.