Do You Actually Need a Local LLM? A Business Guide to Keeping Client Data Private

An accountancy practice spent three weeks pricing a server to run models in-house, because a partner was uneasy about client records going anywhere near a chatbot. While that was being costed, two junior staff were pasting draft management accounts into a free chatbot on their personal logins, from their own laptops, because the firm had not bought them a licence. The expensive project — a local LLM for business, properly specified — was aimed at a threat that was not the one actually happening.

That is the pattern almost everywhere. Running models on your own hardware is a real and sometimes correct answer, but it is the answer to a narrower question than most people think they are asking, and reaching for it first usually means spending four figures to fix a problem that a licence and a one-page policy would have closed in a week. What follows is the order the decision should be made in: what your existing subscriptions already commit to, where the actual exposure sits, the cases where running the model yourself genuinely is the right call, and what that costs once the hardware is on the desk.

What you are already promised

The major vendors publish their business-tier commitments, and they are stronger than the general anxiety around them suggests.

OpenAI’s enterprise privacy page, last updated 8 January 2026, states plainly that it does not train models on your data by default across ChatGPT Business, Enterprise, Edu and Healthcare and the API Platform, that you own your inputs and outputs where law allows, and that Enterprise-tier customers control their own retention period. It lists a completed SOC 2 audit, AES-256 encryption at rest and TLS 1.2 or better in transit.

Anthropic’s position on commercial products — Claude for Work and the API — is that it will not use your inputs or outputs to train its models. The exception is worth knowing because it is the one people trip over: if someone on your team submits feedback, including a thumbs up or down on a response, that conversation can be used for training, and feedback conversations are held for up to five years, de-linked from user and customer identifiers. Team and Enterprise owners can switch feedback off for the whole organisation through the “Rate chats” setting under Data and Privacy, which takes about thirty seconds and is worth doing before you tell staff the data never leaves.

Google’s commitment for Workspace is in the same shape: prompts count as customer data under its Cloud Data Processing Addendum, and the Training Restriction section of the Workspace Service Specific Terms says customer data is not used to train models without the customer’s prior permission or instruction. For licensed Workspace with Gemini users, submissions are not used for training and are not reviewed by humans.

Read those three together and a pattern emerges. The protections are contractual, they are tied to a paid business licence, and they are specific about which product they cover. None of that travels with an employee who opens a consumer account.

The exposure is the account, not the model

Google’s own documentation is explicit about the other side of the line: users without a qualifying Workspace edition fall under consumer terms, where conversations may be human-reviewed and used for product improvement. The same split exists at every vendor. The business tier you negotiated and the free tier your staff signed up for on a Tuesday afternoon are different products with different terms, and the model on the other end does not know or care which one your data is worth protecting under.

So the first move, before any hardware conversation, is dull and cheap:

Licence everyone who would otherwise improvise. A paid seat is the cheapest compliance control you will ever buy. The staff member using a free account is not being reckless; they are being productive with the only tool they were given.

Turn off feedback submission on Anthropic plans if your contract language with clients makes the five-year feedback retention awkward to explain.

Write one page, not a policy binder. Name the three categories of data that must never go into any external tool — for most firms that is identifiable client financials, anything under a specific NDA clause, and credentials — and name the approved tool for everything else. A rule people can remember is a rule people follow.

Check what your own client contracts actually say. Plenty of professional-services agreements restrict disclosure to subprocessors without notice, which is a contractual problem rather than a technical one. This is also the point where the answer sometimes genuinely becomes local, and the rest of this article is about that case.

When running it yourself is the right call

Four situations survive the cheap fixes above.

A contract forbids it. If a client agreement bars sending their material to third-party processors, or requires a named list you cannot amend quickly, no vendor privacy page solves that. The data has to stay on infrastructure you control.

The data is regulated in a way that makes the paperwork disproportionate. Health records, certain legal files, defence-adjacent work. The assessment effort can exceed the cost of a machine in a cupboard.

The work is high-volume, repetitive and boring. Classifying ten thousand support emails, extracting six fields from every invoice, tagging a document archive. These are tasks where a mid-sized open model is entirely good enough, the per-item cloud cost compounds, and nothing about the output requires brilliance. If you are already routing queues automatically, this is a natural fit alongside AI support ticket triage — the classification step does not need a frontier model.

You want to experiment on material you would never upload. Testing prompts against a folder of real client files is enormously more informative than testing against sanitised examples, and a local model lets you do it without a procurement conversation.

Notably absent from that list: general drafting, research, meeting summarisation and the everyday assistant work most teams actually do. For those, the business-tier commitments above already cover you, and local models will be a downgrade.

What a local LLM for business actually costs

The hardware number people quote is the base model, and the base model is not the one that runs anything useful.

A Mac mini starts at $899 with the M6 chip, which is a genuinely capable machine — but at 16GB of unified memory. The configuration that comfortably holds a 27 to 32 billion parameter model with room for context is 32GB, and a Mac mini with 32GB of memory and 1TB of storage lists at $1,699. The M5 Pro tier, which is where you go for 48GB or 64GB, starts at $1,699 before any upgrades. Call it $1,700 to $2,500 for a machine that will do real work, which is roughly two to three years of business-tier licences for a four-person team, paid up front, and it does not replace those licences.

What you can run on it is the better news. Ollama’s library makes the practical options obvious: Gemma 3 at 27B, Qwen 3 at 30B and 32B, Gemma 4 at 26B and 31B, DeepSeek-R1 in sizes from 1.5B up, Llama 3.1 at 8B for lighter work. A 27B-class model quantised to four bits sits in roughly 17 to 20GB, which is why 32GB is the threshold that matters. Getting one running is genuinely a one-evening job, covered step by step in the complete guide to running local LLMs with Ollama, and giving your team a familiar chat window on top of it is a separate short job covered in the Open WebUI setup guide.

The costs nobody budgets for are the other ones. Somebody has to own the machine — updates, model upgrades, the morning it stops responding. There is no support contract. When a model two tiers better is released, you are the one who evaluates and swaps it. For a team with no technical staff at all, that ongoing ownership is the real price, not the $1,699.

Be honest about the quality gap

A 27 to 32B open model running on a desktop is not equivalent to a frontier model, and anyone telling you otherwise is selling something. What it is reliably good at: extracting structured fields from documents, classifying and routing, summarising a single document, drafting from a tight template, and answering questions about text you hand it directly. What it is not good at: long multi-step reasoning, anything requiring current world knowledge, and holding a very large context together coherently.

Match the task to that honestly and local work is unremarkable and useful. Expect frontier behaviour and you will conclude the whole idea was a waste, which is how most abandoned local setups end.

The arrangement most firms actually land on

Almost nobody who runs this properly ends up fully local or fully cloud. They end up splitting by data class, which is less satisfying than a principle and considerably cheaper than either extreme.

A small conveyancing firm is the clearest version. Client files — identifiable, covered by an engagement letter that names its processors — go to a local model for the only thing that touches them at volume: pulling the parties, dates, addresses and title numbers out of scanned documents into a structured record. That is extraction, the task local models are best at, run against the data that must not leave. Everything else runs on a licensed business-tier assistant: drafting correspondence from templates, summarising case law, the general back-and-forth that makes up most of the day. No client file is pasted into it, because the extraction step already produced a de-identified record that is safe to work from.

The split is worth drawing explicitly on a single page, because the failure mode is not technical. It is a person at 4pm who cannot remember which tool the thing in front of them belongs in, and who picks the convenient one. Two tools with a clear line between them get used correctly. Two tools with a vague line become one tool, and it will be the convenient one.

The afternoon that settles the argument

Do not buy hardware to answer this question. Borrow a machine with enough memory, or rent a GPU instance for a day, and run the comparison properly.

Take fifty real documents of the exact kind you would use this for. Write down what a correct output looks like before you see any output — the specific fields, the format, what counts as a failure. Run all fifty through a local 27B-class model, and the same fifty through your existing business-tier cloud subscription. Score both against the sheet you wrote first.

Three outcomes, and all three are useful. If the local model clears your bar, the business case is now a spreadsheet rather than an argument. If it clears the bar on half the task, split the workflow — local for the extraction, cloud for the judgement. If it fails, you have saved $1,700 and can go back to fixing the account problem, which was probably the real exposure anyway. The same discipline applies to anything AI-adjacent you are evaluating: the transcripts produced by AI meeting notes tools are exactly the kind of sensitive archive worth running this test against before you pick a vendor.

Where this goes wrong

Treating local as automatically compliant. A model on your own hardware still sits inside your obligations. Access control, backups, encryption at rest and a retention policy all still apply, and a box under a desk with no disk encryption and no backup is a worse position than a SOC 2-audited vendor, not a better one.

Forgetting that prompts become records. Local chat logs are business records on your infrastructure, discoverable like any other. Decide the retention period on day one rather than discovering three years of them later.

Running it on someone’s working laptop. Inference pins the machine. It needs to be a dedicated box or a server, or people will quietly stop using it.

Nobody patches it. An unmaintained inference server on the office network is an asset for about four months and a liability afterwards.

Buying the hardware first. It remains the most common failure, and it is the one the afternoon test above exists to prevent.

The order is what matters here. Licence your people, write the one page, read your own client contracts, then run the fifty-document test on borrowed hardware. If a local model wins that test, buy the machine with confidence. If it does not, you have lost an afternoon and learned where your data was actually going — which, as the accountancy practice discovered, is rarely where the expensive project assumed.