An AI demo takes a few days. A useful AI system inside a business takes a few weeks longer, and most of that time goes on work that never appears in the demo. When pilots stall, it is almost always for one of five reasons.
1. The data was not ready
The demo ran on a clean sample. Production data is spread across systems, full of duplicates and missing fields, and sometimes nobody is sure who owns it. We start every engagement with a short data readiness review, so the size of the data work is known before the build begins.
A readiness review answers practical questions. Where does the data live, and can we reach it through an API or a database connection? How much of it is complete? Who can approve access? Is any of it personal or confidential, and what rules apply? The answers often change the plan: sometimes the right first step is a small pipeline that collects and cleans data, not the AI feature itself.
2. Nobody agreed what success means
"Make support faster" is not a target. "Answer 40% of order-status questions without an agent, at under 5% error" is. We agree one number with the client in the first week and measure the proof of value against it.
A good success metric has three parts: what is measured, the level that counts as success, and the error or cost you are willing to accept. It should be something the business already tracks, such as handling time, documents processed per day or inquiries qualified, so the result can be compared with how things work today.
3. It lived outside the real workflow
A separate chatbot tab gets used for a week. The same capability inside the CRM, the ERP or WhatsApp gets used every day. Integration is not a finishing touch; it decides adoption.
Design the workflow before the prompt. Who sees the AI output, at what moment, and what do they do next? Can they correct it in one click? Does the correction flow back so the system improves? An assistant that drafts a reply inside the screen where staff already work will beat a more capable one that needs them to switch tools.
4. Quality was checked by feel
Teams try a few questions, like the answers and ship. Then a customer finds the case that fails. We build a set of real test examples before launch and run it after every change, so quality is measured, not guessed.
An evaluation set does not need to be large to be useful. Fifty to two hundred real examples, each with the answer a good employee would give, will catch most regressions. Include the awkward cases on purpose: incomplete documents, ambiguous questions, requests the system should refuse. Run the set whenever the prompt, the model or the data changes, and keep the scores so you can see the trend.
5. Running costs were a surprise
Costs that look tiny in a pilot grow with every user. Model choice, caching, limits and per-customer tracking keep them predictable. We log cost for every request from day one.
Not every step needs the most capable model. Classification and extraction often run well on smaller, cheaper models, with the larger model kept for the steps that need reasoning. Caching repeated questions, trimming the context sent with each request and setting limits per user all keep the bill in line with the value delivered.
The parts of a production AI system
When an AI feature works in production, the model is usually the smallest part of what was built. Around it sit:
- Data access: connectors, cleaning and the retrieval layer that finds the right information for each request.
- Integration: the screens, APIs and messaging channels where people actually use the output.
- Evaluation: the test set, scoring and review workflow that tell you whether quality is holding.
- Guardrails: rules for what the system may answer, what it must refuse and when it hands over to a person.
- Monitoring: logs of inputs, outputs, cost and user feedback, so problems are found before customers report them.
- Security and access control: the same permissions your users already have, applied to what the AI can see.
A realistic timeline
A typical path from idea to production looks like this:
| Stage | Typical length | What you get |
|---|---|---|
| AI use-case | 1–2 weeks | A ranked use case, a data readiness review and the agreed success metric |
| Proof of value | 4–6 weeks | A working version on your real data, measured against the metric |
| Production | Ongoing | Hardening, integration, monitoring and the next use case |
The proof of value is where most of the learning happens. It uses real data, real users and the agreed number, so the decision to scale is based on evidence rather than a demo.
What this means for your first project
Start with one use case, one number and real data. Build a working proof in weeks, measure it, and only then scale. It is slower than a demo and much faster than a pilot that never ships.
A worked example: an order-status assistant
Take a distributor whose customer service team answers hundreds of WhatsApp messages a day, most of them asking where an order is. The demo was easy: paste an order number, get a friendly answer. Production took more thought.
First, data. Order status lived in the ERP, but dispatch details came from three courier portals, and some orders were split across shipments. A small pipeline pulled courier updates into the ERP every hour, which was useful on its own before any AI was added.
Then the workflow. The assistant answered inside the existing WhatsApp number, identified the customer by phone number, and only showed orders belonging to that customer. Anything about pricing, complaints or changes went straight to a person with the conversation attached.
Then quality. The team collected two hundred real messages, wrote the correct reply for each, and scored every change to the prompt against them. The awkward cases mattered most: messages with two order numbers, voice notes, and customers asking in mixed languages.
Finally, cost. Simple status questions used a small, cheap model; only messages that needed interpretation used a larger one. The cost per conversation was logged from the first day, so the business could compare it directly with staff time.
The agreed number was the share of status questions answered without a person. Because it was measured on real traffic, the decision to expand the assistant to payment reminders was easy.
Signs a pilot is about to stall
- Nobody can say which number the pilot is meant to move.
- The data used in the demo is a hand-picked export, not a live connection.
- Users have to leave their normal tools to use it.
- Quality is discussed through anecdotes rather than scores.
- Nobody knows what it will cost at ten times the current usage.
- There is no named owner in the business who will decide whether it goes live.
Questions to ask an AI partner
Whether you build in-house or with a partner, a few questions separate teams that ship from teams that demo. How will you check our data before committing to a timeline? What number will we measure, and how? Where will the feature live in our existing workflow? How will quality be tested after every change? How will we see the cost per request? Who supports it after launch? Clear, specific answers are a good sign; promises about the model alone are not.
Frequently asked questions
How long does it take to get an AI feature into production?
For a well-chosen use case with accessible data, a working proof usually takes four to six weeks, and production hardening follows from there. Data access is the most common cause of delay.
Do we need our own data scientists?
Not to start. Most business use cases are built by software and data engineers using existing models, with evaluation and monitoring around them.
Which model should we use?
The one that meets your quality target at an acceptable cost and fits your privacy needs. Keep the model behind a thin layer in your code so you can switch as models and prices change.