Local AI or cloud AI: what a small business actually gains

Every few weeks the same pitch lands in your inbox: "with us, the AI runs on your own server — your data never leaves the building." It works, because there's a reasonable fear underneath it. You don't love the idea of your customer conversations sitting on the infrastructure of a company eight time zones away. The trouble is that "on-premise" doesn't mean what most people think it means, and the decision that actually matters isn't the one you're being sold. Here are the three real options, what each one costs, and when the complicated one is worth it.

There are three options, not two

Cloud. You send a message to a provider — OpenAI, Anthropic, Google, Mistral — the model runs on their infrastructure and the answer comes back. You pay per use and maintain nothing.

Actually local. The model weights (the enormous file that is the model) sit on a machine you own: the office desktop, a small server in the cupboard next to the router. Nothing goes out to the internet.

The middle option nobody mentions. An open-weight model running on a server you rent in a data centre in Madrid, Frankfurt or Paris. Still a third party, yes — but a European one, on a contract you signed, without a long chain of sub-processors, and with nothing crossing the Atlantic. For most small businesses flirting with on-premise, this is the option they were actually looking for.

Lumping all three together is what turns this into an ideological argument. It isn't one. They're three cost structures and three risk profiles, and you pick per process, not per belief.

Where the money actually goes

Cloud is billed per message processed, in fractions of a cent, and drops to zero when you're not using it. Pure variable cost: quiet August, small invoice.

On-premise is paid up front and then looks free. It isn't. You need a machine with enough RAM for the model: the docs for Ollama, the usual tool for this, recommend having at least 8 GB of RAM available for 7-billion-parameter models, 16 GB for 13B, and 32 GB for 33B. Add a decent GPU unless you enjoy ten-second pauses, add electricity, and add the big one — somebody who patches it, watches it, and restarts it when it hangs.

Do the back-of-the-envelope yourself: take the price of the machine, divide by 36 months, add the hours a month someone will spend on it valued at what that person actually costs you, and compare against the same volume through an API. At small-business volume — hundreds of messages a day, not millions — the sum lands on the cloud side almost every time, and not narrowly. Hardware isn't the expensive part. Somebody's attention is.

"On-premise" is not the same as "GDPR-compliant"

This is where most people go wrong. GDPR applies to you wherever the model lives: you still need a lawful basis, you still have to tell people, you still keep only what you need for as long as you need it, and you still have to honour a deletion request.

What changes with cloud is that the provider becomes a processor, which requires a contract with them. And if data leaves the European Economic Area, you need cover for that: an adequacy decision, standard contractual clauses, or one of the other safeguards in the regulation. Spain's data protection authority sets out the options in its FAQ on international transfers. None of this is exotic — serious providers hand you the paperwork ready to sign — but it exists, and someone has to read it.

Where local genuinely wins isn't abstract "privacy". It's three specific things: the international transfer question disappears, the sub-processor chain disappears, and you can work offline. If a client has written into your contract that their documents don't leave your company, no clause fixes that. Not sending them fixes it.

And where local saves you from nothing: a model running on an unencrypted machine with no backups and the password on a sticky note is worse for your customers than an API with a signed contract. Data being close by is not data being protected.

The quality gap is still real

Open models small enough to run on an office machine have improved enormously, but they don't do what the large ones do. What they handle well:

• Classifying: is this email a quote request, a complaint, or spam?
• Pulling structured data out of text: name, phone, registration number, amount, date.
• Compressing a long message into two lines.
• Tagging and routing: which folder, which person on the team.

Where the seams show: long conversations with nuance, holding fifteen business rules at once without dropping one, switching language mid-sentence, catching the sarcasm in a message from an annoyed customer. If the agent is going to talk to your customers directly on WhatsApp, you'll notice the difference in week one.

The honest test is easy, and no salesperson will run it for you: take fifty real messages of your own — including the weird ones, not the tidy ones — and put them through both. The demo always works. Your fifty messages don't always.

When not to build an AI server

Four fairly reliable signs this will cost you more grief than it saves:

1. Nobody can restart it at eight on a Friday evening. If the answer to "who fixes it when it falls over?" is you, you already know how this ends.
2. Your volume is low. Below a few thousand messages a month, the payback never arrives.
3. You're doing it "for privacy" but haven't read your current provider's contract. Start there: what did you sign, where is data processed, is it used for training? The problem you're solving may not exist.
4. You need conversation, not classification. If the job is talking to people, don't hand it to the small model just because the machine is yours.

A system that works 95% of the time and that nobody can fix the other 5% isn't a system. It's a breakdown with a date on it.

The boring hybrid that actually works

Almost no small business needs to pick a side. What works is splitting by data sensitivity, and it's an afternoon's decision:

1. Write down what data the process touches. An actual list: names, phone numbers, records, account details, client documents.
2. Mark the ones that can't leave your company, by law or by contract. It's usually far fewer than you'd guess.
3. For those specific steps: small local model, or no AI at all.
4. For everything else: cloud with a processor agreement, EU processing, and a written commitment that your data isn't used for training.
5. Send the minimum. Nobody needs a customer's full history to draft "your order ships tomorrow."

Point 5 protects you most and gets done least. The real question was never "cloud or local." It's how much you tell whom, and why.

If that's where you are, order matters: decide which process you want off your plate first, then decide where it lives. Our operations agent handles the back-office — quotes, follow-ups, reminders — and where each step gets processed is a decision we make with you, not a default. You approve, it executes.