Scenarios where local inference is the only option
AI with local and open models · Lesson 2 / 21
Some tasks have made the choice for you already
Local inference is usually compared with the cloud on money and convenience. But there is a class of tasks where no comparison happens at all: a cloud call is either technically impossible or forbidden by the rules, and the question narrows to how you fit a model into the resources you have. Recognise these scenarios early, because they change the whole design process. You are not picking the better option, you are taking the only one available.
Data that must not leave
Personal, medical and payment data, the contents of employee correspondence, source code under NDA, classified documents. The constraint here is rarely the act of transmission itself; it is the contractual and regulatory wrapper around it. The processor must be on an approved list, the site must sit in the right jurisdiction, the retention period must be under your control. A provider may formally satisfy all of that, but the approval takes months and sometimes ends in a refusal from the security team of the counterparty who owns the data.
A network that is not there
- Isolated segments. An industrial network, a closed segment, a test bench with no route outside. No proxy to the internet will appear there, because its absence is the security control.
- Equipment in the field. A rig, a ship, a vehicle, a remote site: there is a satellite link, but it is narrow, expensive and it drops.
- The user's own device. Processing right on the laptop or phone, when the data must not even reach your own server.
Hard latency budgets
A cloud call adds a network path to generation: the handshake, the route to the provider's region, the queue on their side. On a good link that is tens of milliseconds, on a bad one hundreds, and the distribution has a long tail. If your scenario demands a fixed response time in the upper percentile — inline completion, a voice loop, a reaction to a real-time event — that tail becomes the deciding factor. A local model removes the network from the equation entirely, and only generation time remains, which you control.
Volume that does not get through
signal in the scenario consequence -------------------------------------------------------- steady request flow hardware stays busy, idle time low short, uniform requests a small model is enough short responses generation time is predictable internal data no approval process needed -------------------------------------------------------- all four present - local is almost certainly the right answer
Ticket classification, labelling a stream of documents, extracting fields from forms, filtering logs — here the requests look alike, the flow is steady, and the provider's requests-per-minute limits become a wall long before the budget runs out.
Cheat sheet
- Some scenarios make a cloud call impossible; recognise them before choosing a model.
- Four typical ones: restricted data, an isolated network, a hard latency budget, a steady high-volume flow.
- The network adds a long tail to latency, and local inference removes it completely.
- Locality covers transmission, not the whole data lifecycle.
Describe one real scenario of your own and check whether local inference is forced for it or merely preferable. Write down four constraints: which data is involved and who must not see it, whether the environment has internet access and of what quality, the maximum latency allowed at the 95th percentile, and the request flow expected at peak and on average. For each item, state where the number came from — a measurement, a document, or an assumption. Finish with a verdict: the cloud is impossible, the cloud is possible but awkward, or the cloud is fine.