Autopsy Before Autopilot: A Field Checklist for AI Pilots That Will Never Ship
Photo: TheAHL, CC BY 2.0, via Wikimedia Commons
Every enterprise technology survey tells the same story: the vast majority of AI pilots never reach production. The reasons cited are usually technical — data quality, model drift, integration complexity. But after reviewing dozens of stalled initiatives across banking, insurance, and manufacturing, a different pattern emerges. Most doomed pilots were identifiable as doomed in their first month, based on decisions that had nothing to do with machine learning.
Here is a diagnostic checklist. If your current AI initiative checks three or more of these boxes, you are not running a pilot; you are funding a burial.
The Seven Warning Signs
1. No one owns the P&L impact. If you cannot name the specific person whose budget improves when this works, the project has no constituency. Survivors tie every pilot to a named operator — a collections manager, a claims lead — not to an innovation team.
2. The use case was chosen for impressiveness, not pain. Generative demos win board applause; they rarely win adoption. Pilots that ship target a workflow so tedious that employees are already improvising workarounds.
3. The data owner is not in the room. If the team building the model had to 'request' access to the data it needs, and that request is still in a queue, the pilot is dead on arrival. Data governance participation must be a founding condition, not a later milestone.
4. Success is defined as 'the model performs well.' Model metrics are not business outcomes. Survivors define success in operational terms from day one: handling time reduced by 20 percent, false-positive rate below a threshold a human reviewer can sustain.
5. There is no fallback workflow. A pilot without a defined path for what happens when the model is wrong — and it will be wrong — cannot safely enter production. Teams that scale AI design the human escalation route before they tune the model.
6. The timeline exceeds two quarters. Long pilots accumulate stakeholders, and stakeholders accumulate requirements. By month nine, the original problem has been redefined twice and the sponsor has moved on. Speed is not a nice-to-have; it is the mechanism by which learning happens.
7. Legal and compliance first hear about it at launch. In regulated US industries — healthcare, financial services, anything touching employment decisions — late involvement of counsel is the single most common cause of shelved projects. Early involvement feels slower and is dramatically faster.
What the Survivors Do
The pilots that reach production are rarely the most technically ambitious. They share a discipline: they start smaller than anyone wants to admit. One call center, one document type, one region. They measure against a boring baseline — the current manual process, not perfection. And they treat the model as one component in a workflow, alongside the humans, approvals, and audit trails that surround it.
They also invest early in the unglamorous layer: logging, evaluation datasets, and rollback plans. This is the AI equivalent of unit testing — nobody celebrates it, and everybody benefits from it.
Run the Checklist Now
The most valuable moment to apply this diagnostic is before the next budget cycle, not after. Gather the team, score the pilot honestly against the seven signs, and have the courage to kill what fails. A shelved project costs more than money; it consumes organizational credibility that the next, better-designed initiative will need. In enterprise AI, knowing what not to build is a competitive advantage.
