Local AI for Companies: When It Is Really Worth It

"The data can't leave." If someone in your company has said it, this pill is for you. In a few minutes you'll know when a local AI model is worth it, which tasks it handles, what it really costs, and how to decide, task by task, what stays in-house.

What It Is

A local model is an AI model running on a system under your control: a server room, a private cloud, a dedicated GPU. The model weights (the files that make it work), the logs and the documents passing through stay inside the perimeter you signed. It's not a laptop under a desk.

Two variants not to confuse. On-prem means in your building. Self-hosting can also sit on a European cloud, with your own keys. For some clients the difference is legal, for others it's just speed. Put it in the contract up front.

Why It Matters

Sometimes the constraint comes from legal, sometimes from a public-sector client, sometimes it's just fear. Either way it's a project constraint, not an opinion about which model is best.

The answer isn't "open versus closed models". It's a fleet: some work stays in-house, some goes to a frontier model (the most powerful models, used through an API) with a contract and a DPA, the data processing agreement. A router, a system that dispatches, decides task by task.

In our own test we put Mistral 7B and Qwen 2.5 7B into production on on-prem A10G GPUs, on three tasks: document classification, structured data extraction and internal drafts. Compared with Sonnet 4.6:

• quality: 92%, in a blind comparison on a sample labelled by the client;
• latency: down 65%;
• cost per million tokens: from €3.10 to €0.12, roughly 1 to 26.

92% means eight more errors per hundred, on tasks where errors are visible and fixable. Not on an expert report.

Where Local Holds Up

• Classification: a document goes in, a label comes out (contract, invoice, complaint, other). High volume, narrow judgment. A small model instructed on your labels beats a large one that has never seen them.
• Extraction: fields from a delivery note, a policy, a set of minutes. Fixed schema, measurable errors. Here local is good enough and much cheaper.
• Internal drafts: team summaries, first-draft emails, minutes. They cost so little you do them every time, not just "when there's time". Nothing goes to a client without review.

Outside these, local loses today: fine legal analysis, architecture, new code in an unfamiliar domain. That's where the frontier model goes. It costs more and fails less. For many enterprise clients, the share of work that deserves it sits below 20%.

Italian follows the same line. On our set of 240 Italian prompts (tax, logistics, regulations), open models stumble on terminology and nuance. The line isn't "Italian versus English", it's closed schema versus open language. A delivery note is a closed schema: it stays in-house. An opinion on tender specifications is open language: today, it doesn't.

How to Do It

1. List tasks, not wishes. "Classify this", "extract that", "draft this", "answer this hard question".
2. For each, estimate volume and cost of error. High volume, low harm: local. Low volume, high harm: frontier. High volume and high harm: don't start with AI, start with the process.
3. Build a labelled sample. Even two hundred examples. Without a sample, 92% is just talk; with one it becomes a threshold: below it, nothing goes to production.
4. Add a router. The team sends the task, the router picks the model. Nobody has to "remember" which one to use.
5. Log where each task goes. A router that records "this went to the local model, this to the frontier" is already part of the answer to the AI Act, which asks you to know what the system does.
6. Decide who keeps it running. You need a person who updates the weights, watches the logs and unblocks a stuck queue. If there isn't one, self-hosting on a European cloud with a clear contract is more honest than on-prem.

Ready to copy. Paste into Claude or ChatGPT the list of tasks you'd like to hand to AI (task names only, not the documents):

Try It Small

Pick a closed-schema task: for example, twenty documents to label as contract, invoice, complaint or other. Write the correct label yourself. Then have a small model label them and count how many it gets right. In ten minutes you have a first number of your own, not ours. Until the model runs in-house, use fake or anonymised documents.

Watch Out For

• €0.12 is the inference cost, not the total. Add GPU depreciation, electricity and the hours of whoever runs it. At high volume it's still cheaper than the frontier, but it isn't free. Anyone selling "local AI at zero cost" is leaving out the staff line.
• Local doesn't mean private by magic. It is private if nobody exfiltrates prompts, logs rotate and access is named. Otherwise you've just moved the risk into a room you know less well.
• Batch and front desk are two different projects. A nightly classifier runs in batches, with a queue, retries and a morning report. A front-desk operator needs the answer now: the 65% latency cut applies to that case. Don't mix them.
• Don't buy a GPU and send it everything. You'll waste power and quality.
• Don't run two worlds that never talk, ChatGPT for managers and Ollama for IT. It's the same work split in two.
• Don't promise legal that "nothing leaves" if some reasoning has to leave. Say so, put the DPA in place, instruct the router. Trust lives in the map, not in the ban.

If data can leave and the problem is spend, read how to stay within your AI plan limits. If you want to draw the line and put it into production, get in touch: the system stays yours.

In short: Local AI pays off on high-volume, repetitive work with a closed schema; hard reasoning stays on a frontier model, and the line is drawn task by task.