Local LLMs for repetitive enterprise workloads

Many Italian companies arrive with the same sentence. «The data cannot leave.» Sometimes it is legal. Sometimes a public-sector client. Sometimes fear. In every case it is a project constraint, not an opinion about the best model.

Open versus closed does not help. You need a fleet. Some traffic stays in-house. Some goes to a frontier API with a contract and a DPA. The router decides per task, not per flag.

We put Mistral 7B and Qwen 2.5 7B into production on on-prem A10G GPUs. Document classification, structured extraction, internal drafts. On those three workloads, observed quality is 92% of Sonnet 4.6. Latency −65%. Cost per million tokens from €3.10 to €0.12. Complex reasoning stays outside, on the frontier.

What we mean by «local»

Local is not a laptop under a desk. It is a runtime under your control. Server room, VPC, dedicated GPU. Model weights, logs, documents: they stay in the perimeter you signed.

Self-hosting and on-prem are not the same. Self-hosting can sit on EU cloud with your keys. On-prem sits in the building. For some clients the difference is legal. For others it is latency. Say it in the contract first.

A local model is not «more private» by magic. It is private if prompts are not exfiltrated, logs rotate, and access is named. Otherwise you only moved the risk into a room you know less well.

The three workloads that hold

Classification. A document in, a label out. Contract, invoice, complaint, other. Volume is high. Judgement is narrow. A 7B instructed on your schema beats a large model that has never seen your labels.

Extraction. Fields from a delivery note, a policy, a minute. Fixed schema. Measurable errors. Here local is good enough and much cheaper. 92% is a blind comparison on a client-labelled sample, not a slogan.

Internal drafts. Team summaries, first-pass mail, minutes. No text leaves the perimeter to an external client without review. Low cost means you do them always, not only «when there is time».

Outside these three, local loses today. Fine legal analysis. Architecture. New code on a domain the operator does not have in hand. There we send the frontier. It costs more. It fails less.

The numbers, without rounding

Quality 92% on those workloads versus Sonnet 4.6. Eight extra errors per hundred, on tasks where the error is visible and correctable. Not on an expert report.

Latency −65%. The internal user does not wait on an overseas API queue. For an operator classifying fifty documents a day, that wait is the job.

Cost from €3.10 to €0.12 per million tokens. Roughly 1 to 26. If you classify hundreds of thousands of pages a year, frontier on the whole volume is a tax. If you make ten hard reasoning calls a day, frontier is the right price.

Our router treats local and cloud as one fleet. The boundary moves per task. For many enterprise clients, the share that deserves frontier sits below 20%.

EU data, contract, AI Act

«Data in Europe» is not a slide. You need the processor, the region, who can read logs, how long they live.

A commercial LLM with a clear DPA remains the right call on sensitive reasoning work, or when self-hosting is immature. We wrote that in the Italian technical benchmark too. Local does not replace that contract. It sits beside it.

The AI Act asks you to know what the system does. A router that records «this task went to local Qwen, this one to Sonnet» is already part of the answer. One opaque model, in-house or not, is not.

How to draw the line

List tasks. Not wishes. Classify this. Extract that. Draft this. Answer this hard question.

Measure volume and cost of error. High volume, low harm: local. Low volume, high harm: frontier. High volume and high harm: do not start with AI. Start with the process.

Build a labelled sample. Even two hundred examples. Without a sample, 92% is talk. With a sample it is a threshold: below it, no production.

Then put in the router. Do not ask the team to remember which model to use. The team sends the task. The router chooses.

What not to do

Do not buy a GPU «to have AI in-house» and then send it everything. You will waste power and quality.

Do not keep two stacks that do not talk: ChatGPT for managers, Ollama for IT. That is the same work split into two shadows.

Do not promise legal that «nothing leaves» if some reasoning must leave. Say it. Put the DPA in place. Instruct the router. Trust lives in the map, not in the ban.

If your documents cannot travel, or volume makes frontier unsustainable, local is a fleet tier. Not an act of faith. We can help you draw the line and put it into production. The runtime stays yours.