Why technology forecasts fail in both directions

The future of AI and AGI: an overview · Lesson 1 / 20

A failed forecast is a pattern, not an accident

People talk about the future of AI as if the argument were between those who are right and those who are wrong. In practice almost everyone is wrong, and wrong in predictable ways. Understanding the mechanics of those errors is more useful than hunting for the correct prophet: the mechanics repeat, the prophets change.

There are two symmetrical errors. The first is overestimation: someone sees a technology as a working demonstration and mentally completes it into a finished product that will sit in every company by tomorrow. The second is underestimation: someone sees a raw, faintly ridiculous first version and concludes it will stay that way. Both errors come from the same substitution — reasoning about impressions instead of reasoning about mechanisms.

Where overestimation comes from

Extrapolating the curve. Because improvement was large over a few steps, people conclude the next steps will be just as large. But a curve is the result of specific mechanisms at work, not a thing in itself; when a mechanism hits a bottleneck, the curve changes shape.
Demonstration instead of deployment. Between the system can do this on a prepared example and the system does this reliably for other people on their data lies most of the work. That work is usually invisible from the outside.
Ignoring everything alongside. Technology does not work in a vacuum: it needs data, integrations, trained people, acceptance procedures, someone accountable when it fails. This surrounding structure develops more slowly than the model does.

Where underestimation comes from

Judging by the current version. An early version is graded as if it were final, although its weaknesses may be fixable or may be fundamental. Those are different cases and they need to be told apart.
Betting on real quality. A technology often wins not by doing the work better than a person, but by doing it well enough and far more cheaply.
A blind spot for side effects. The most visible consequences of a technology are usually not the ones it was built for.

Taking a forecast apart
1. Mechanism - what exactly keeps the improvement going
2. Bottleneck - what runs out first: data, money, energy, staff, trust
3. Condition - under what circumstances the claim holds
4. Observable consequence - what we see if the forecast comes true
5. Refutation - what we see if it does not

Insight. A forecast with no item 5 is not a forecast, it is a mood. A claim compatible with every outcome cannot be checked, so nothing can be derived from it.

Common mistake. Assuming caution is always the more accurate position. Systematic scepticism is wrong exactly as often as systematic enthusiasm; its errors simply show up later and get recalled less often.

Pro tip. Write down your expectations with the date you wrote them and a stated observable consequence. Six months on, that is the only way to learn how well you read the field - memory quietly edits past opinions to fit what happened.

Separate three different claims

A single sentence often blends a claim about capability (the system can do this), a claim about diffusion (this will be used everywhere) and a claim about consequences (because of this, that will change). They demand completely different evidence and resolve on different timescales. Capability is checked by reproduction. Diffusion is checked by economics and regulation. Consequences are checked by watching real organisations. An argument where one side talks about capability and the other about diffusion cannot end: the two people are discussing different things.

Cheat sheet

Overestimation and underestimation are two faces of one error: judging by impression instead of by mechanism.
Every growth curve has a mechanism and a bottleneck; ask about both.
A forecast with no stated refutation is untestable and therefore useless.
Capability, diffusion and consequences are three separate claims with separate evidence.

1. Why is extrapolating a curve of improvements a weak argument?

2. What separates a testable claim about the future from an untestable one?

3. Two people argue: one says the system can do this, the other says it will not be everywhere. What is wrong with the argument?

🔒 Answer the question correctly to move on to the next lesson.