Telling a real opportunity from a presentation

AI strategy for executives · Lesson 2 / 20

What the demo shows and what production will look like

The demo always works. It runs on data prepared by the vendor engineer, on three favourable examples, with no integration into your systems. The gap between demo and production is not about model quality. It is that real input documents arrive in six formats, half the fields are empty, and the output must land in a system of record with no usable interface. Your job in the vendor meeting is not to evaluate technology but to work out who closes that gap and at whose expense.

Five questions that change the conversation

Whose data is the demo built on. If it is synthetic or from another client, ask them to repeat it on your extract for a week that you choose.
How many of your deployments reached production. Not how many pilots, but how many still run a year later. A healthy market ratio is about one in three; if you hear nine in ten, ask for their definition of deployment.
What the system does when it is not confident. No fallback mode means errors will flow into the process silently.
Who on our side has to work, and how much. An answer like assign an analyst for a couple of hours a week means the vendor has not costed your share of the work.
What happens if this becomes a free platform feature within a year. The reaction tells you more than the preceding hour did.

Red flags at an initiative review
— effect computed against total company revenue rather than a site
— no process owner on the project team, only IT and the vendor
— no description of behaviour under low confidence
— first measurable result promised beyond 90 days
— no answer on what replaces the solution if the vendor disappears

Insight. The best predictor of failure is who shows up to the first meeting. If IT and procurement attend but the process leader does not, the odds of reaching production drop sharply: nobody will rewrite the procedure or argue with their own people.

Common mistake. Judging vendors by model quality. Models are comparable and change every few months. The difference lies in integration, support, and willingness to be accountable for a result inside your process.

Pro tip. Ask for a pilot on your worst data. If the solution holds on the bottom quartile of input quality, it holds everywhere else. The reverse is not true.

Cheat sheet

A demo tests the model; production tests the process and the data.
Ask what share of pilots survives a year, not how many pilots there were.
No process owner on the team is an early failure signal.
Require the first measurable result within ninety days.

1. A vendor demonstrates the solution on their own data. What is the most informative next step?

2. Your side sends the IT director and procurement; the process owner is absent. What does that mean?

3. A vendor claims nine of ten pilots reached production. A sensible response?

Task — checked by AI

Take one AI proposal currently on your desk, or the last one you were shown. Answer the five questions from the lesson in writing for that proposal: whose data the demo used, how many of the vendor's deployments are alive after a year, what the system does under low confidence, how many hours of your own people it needs per month, and what replaces the solution if the vendor disappears. For each point, state whether you have an answer or not. Finish with an explicit decision: proceed, pause until answers arrive, or decline.

← Back

🔒 Answer the question correctly to move on to the next lesson.