Adnan Obuz has sat in more AI pilot reviews than he can count, and the ending is almost always the same … the demo works, everyone claps, and then nothing ships. Eight months later there’s a slide deck with a screenshot of a chatbot answering one question correctly. That’s it. That’s the whole return on the investment.
So this isn’t a technology problem. The models work fine. It’s a structural problem, and it’s fixable. Here’s how Adnan Obuz breaks it down for leadership teams in Toronto and across North America.
Why AI pilots stall, according to Adnan Obuz
The short answer? Companies run pilots as demonstrations, not as the first slice of a production system. And demonstrations don’t have a natural path to production. They have a natural path to a follow-up meeting.
The numbers back this up. Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, pointing to poor data quality, weak risk controls, rising costs, and unclear business value. Adnan Obuz sees all four in the field. Constantly.
The five signs Adnan Obuz looks for in a stalled pilot
Look, the symptoms are consistent enough to work as a diagnostic:
- Nobody can say what percentage of real cases the demo handles correctly, because nobody ever defined “correctly.”
- The pilot runs on a personal API key or a department credit card.
- An innovation team sponsors it, not the function that owns the actual workflow.
- Security and privacy review hasn’t started, since everyone calls it “just an experiment.”
- Nobody has turned the monthly token bill into a cost per completed task.
If four of those five are true, the pilot won’t ship. It’ll get extended.
Adnan Obuz remembers one review in particular. A mid-market insurer had a claims assistant that looked flawless in the boardroom. Then someone asked how it handled a scanned form with a coffee stain across the policy number. Silence. Nobody had tested a single ugly case. That one question killed six months of momentum.
Fix 1: Adnan Obuz says give the workflow an owner, not just the model
Most pilots belong to whoever got curious first … a data science lead, a CTO’s direct report, a digital transformation team. That person owns the model. Nobody owns the end-to-end workflow the model is supposed to change.
And that matters, because production AI is almost never a standalone feature. A claims-triage assistant changes how adjusters queue work, how they escalate exceptions, what gets audited, and what the QA sample looks like. If the VP of Claims doesn’t own it, none of those downstream changes happen. The assistant becomes a side tool that busy staff politely ignore.
The fix is boring, honestly. Name a business owner who already controls the process, the headcount, and the service metric the AI should move. If nobody will sign up? That’s useful information. According to Adnan Obuz, it usually means the team picked the use case for its demo value, not its operational value.
How Adnan Obuz tests for real ownership
He asks one question: “If this system gets something wrong on a Tuesday afternoon, whose phone rings?” If the room goes quiet, there’s no owner. There’s just a sponsor.
Fix 2: According to Adnan Obuz, measure it instead of trusting the vibe
Ask a team how they know their pilot works and you’ll usually hear some version of “we tried it and it seemed good.” That’s a fine reason to keep experimenting. It’s a terrible reason to put a system in front of customers.
Production needs a golden evaluation set. That’s a fixed, versioned collection of real inputs with agreed-upon acceptable outputs, built by the people who actually do the work. In Adnan Obuz’s experience, 150 to 300 examples is the practical range for one workflow. Below about 100, you can’t tell a real regression from noise. Above 400, the labeling work drags on so long it never gets finished.
The set should include the ugly cases on purpose … the ambiguous request, the customer who asks three things at once, the blurry scan, the question in a second language. Pilots get judged on the median case. Production risk lives almost entirely in the tail.
Once that set exists, every prompt change, model upgrade, and retrieval tweak gets scored against it before release. It’s not glamorous. It’s the single highest-leverage habit Adnan Obuz recommends, and the one teams skip most often because it feels like overhead during the exciting phase.
The NIST AI Risk Management Framework makes a similar case for ongoing measurement across the full life of a system. It’s worth a read if your board wants an outside reference point.
Fix 3: Adnan Obuz on modeling the real economics
Token prices have dropped hard, and that’s actually made teams sloppier about unit economics, not more careful. The number that matters isn’t cost per thousand tokens. It’s the fully loaded cost per resolved task … retries, human review time, escalations the system created, and the engineering hours needed to keep retrieval sources fresh.
Adnan Obuz watched one document-processing pilot look amazing at three cents a document. Then it fell apart at scale because 18 percent of outputs needed a senior reviewer, and that reviewer’s time was the real cost all along. The model was cheap. The exception path wasn’t.
Run the math before you scale. A simple version works: expected volume, automation rate at your accepted quality bar, human handling time for the rest, and the blended hourly cost of the people doing that handling. If the answer isn’t clearly positive at realistic automation rates … not the optimistic ones … the workflow isn’t ready yet.
For more on where these budgets quietly leak, Adnan Obuz covers it in costly AI pitfalls Canadian companies keep making.
The production operating model Adnan Obuz uses
Choose workflows, not use cases
“Customer service” is a use case. “Drafting first replies to billing disputes under $500, routed from the shared inbox, reviewed by a tier-one agent before sending” is a workflow. Only the second one can ship, get measured, and have an owner. Narrow scope isn’t timid. It’s what makes accountability possible.
Design the human handoff first
Before anyone writes a prompt, decide what happens when the system is unsure or wrong. Who sees it? What does their screen show? Is the AI output a draft, a suggestion, or a decision? Teams that design the happy path first and bolt on exception handling later end up shipping systems that break trust in exactly the wrong places.
Bring privacy in at week one
In Canada, that means getting clear early on where personal information flows and how PIPEDA rules on consent, retention, and access apply to inputs and outputs. Federal AI-specific legislation has had a stop-start history, but the privacy law already on the books can sink a project discovered late. A 30-minute conversation in week one beats a legal review in month nine. Every time.
Instrument before you scale
At minimum, log inputs, outputs, model version, latency, cost, human override rate, and the reason for each override. That last field is gold. It tells you, in the words of the people doing the work, exactly where the system fails and why … which is basically your roadmap for the next round.
A 90-day sequence Adnan Obuz trusts
Days 1 to 15: Pick one workflow. Name a business owner. Baseline today’s performance … volume, handling time, quality, cost. If you can’t baseline it, you can’t prove it got better.
Days 16 to 35: Build the evaluation set with the people who do the work. Map the target workflow, including the exception path. Run privacy and security review on the design, not the finished product.
Days 36 to 60: Build to the evaluation set. Iterate on retrieval, prompts, and the interface until you hit the quality bar you set in advance … not the one you discover you can reach.
Days 61 to 90: Run in shadow mode, then a limited live rollout with monitoring. Measure against the day-15 baseline. Make a real go, adjust, or stop call.
Ninety days is enough for one workflow. It’s not enough for an “AI transformation program,” and companies that promise that in 90 days usually deliver neither. That’s the honest constraint here.
What this means for your company
The companies pulling ahead right now don’t have better models. Frontier capability is turning into a commodity your competitors can buy on the same terms you can. The lasting edge is organizational … the ability to take a capability, wrap it in evaluation, ownership, economics, and exception handling, put it into daily use, and then do it again next quarter.
That’s a repeatable muscle, not a one-time launch. Teams that build it will ship their fifth AI workflow while competitors are still arguing about their first. The gap compounds. The OECD AI Principles frame accountability the same way, as something you build into how an organization runs.
If your pilots have been “almost ready” for several quarters, the bottleneck probably isn’t the technology. It’s the operating model around it. Adnan Obuz digs into the partnership side of this in three non-negotiables for every AI strategy partnership, and into the decision side in the AI confusion trap.
Frequently asked questions
Who is Adnan Obuz?
Adnan Obuz is a Toronto-based AI strategy consultant who helps mid-market and enterprise leadership teams move AI from pilot projects into dependable daily operations. His work centers on workflow selection, evaluation design, unit economics, and governance.
How do you know if an AI pilot is ready for production?
Adnan Obuz uses three tests. Is there a named business owner who controls the workflow? Can you state accuracy against a fixed evaluation set of real cases, including hard ones? Does the fully loaded cost per resolved task beat your current baseline at a realistic automation rate? If any answer is no, you’ve got a promising experiment, not a production candidate.
How big should an AI evaluation set be?
For a single workflow, 150 to 300 real examples usually works. Below about 100, you can’t reliably spot regressions. Above 400, teams rarely finish or maintain the set. What goes in it matters more than size, so include messy, ambiguous, and edge cases on purpose.
Should a company build AI in-house or buy vendor tools?
Buy the commodity, build the differentiator. Transcription, general document extraction, and coding assistants are mature vendor categories. Build where the value comes from your own data or a workflow competitors can’t copy. Either way, keep evaluation and vendor monitoring in-house, or you can’t tell whether a vendor is delivering or just invoicing.
By Adnan Obuz, AI Strategy Consultant. Adnan Obuz advises leadership teams in Toronto and across North America on turning AI experiments into systems that actually run the business. He focuses on the unglamorous mechanics that decide whether AI spending pays off: workflow selection, evaluation, unit economics, human handoffs, and Canadian privacy obligations. Read more from Adnan at mrobuz.com or reach him at businessplan@mrobuz.com for advisory work and speaking.