Insights / Production AI draft — review before publish
Why pilots work and production systems fail
Every failed enterprise AI initiative I have examined shares a shape: a pilot that genuinely worked, followed by a production effort that quietly stalled. The usual diagnosis — “the model wasn’t good enough” — is almost never the real one.
Pilots succeed because they are built in a vacuum that flatters them. Clean sample data. A friendly user who knows what to ask. No permissions model, no audit requirement, no latency budget, no cost ceiling, no integration with the systems where the work actually lives. The pilot answers the question “can a model do this?” — and that was never the hard question.
Production asks different questions:
Where does the data actually come from? Not the curated sample — the live feed, with its missing fields, format drift, duplicate records, and the upstream system nobody wants to touch. Most “AI projects” that stall are data-integration projects wearing a costume.
Who is allowed to see what? The pilot ran on one person’s access. Production inherits the organization’s entire permission structure, and retrieval systems that ignore it are security incidents waiting to be scheduled.
What happens when it is wrong? Every AI system is sometimes wrong. Production design is deciding — per decision, by consequence — whether wrong means a human reviews it, a rule overrides it, or a customer is harmed by it. Systems that answer this with a disclaimer are demos.
How do you know it is still good? A pilot is judged once, by impression. A production system is judged continuously, by measurement, against drift in data, models, and usage. If there is no evaluation harness, there is no quality — only anecdotes.
What does it cost at volume? Inference economics that are invisible at pilot scale become the P&L at production scale. Architecture — caching, routing, model tiering, deterministic short-circuits — is where the unit economics are actually set.
The pattern behind all five: models are one component. Production AI is a systems problem — data, integration, security, evaluation, economics, and accountability, arranged so the model’s capability survives contact with the organization.
That is why the gap between a working pilot and a working system is not a bigger model. It is architecture, and the judgment to know what should be deterministic, what should be generative, and what should stay human.