Loading…
Loading…
A self-hosted LLM keeps prompts and data on your own servers. When it pays off, what it really costs, the hardware it needs and how to run it safely.
A self-hosted LLM runs on infrastructure you control, so prompts, documents and answers never pass through a model provider.
The case is strongest for regulated data, air-gapped sites, high and steady volume, and a model version you choose and keep.
GPU memory is the binding constraint. Hugging Face puts the weights alone at roughly 2 GB per billion parameters at 16-bit precision.
Read the licence before the benchmark. Open models come with very different terms, and some carry conditions that matter at enterprise scale.
Self-hosting moves the security work to you. Build against the OWASP Top 10 for LLM Applications, starting with prompt injection.
Do not self-host at low volume, without a team to run it, or when the task needs the very largest frontier models.
Search for self hosted LLM and the first page belongs to Reddit threads, Medium posts, hobbyist blogs and hosting vendors. Most explain how to get a model running on a laptop in an afternoon. That is the easy part.
The questions that decide an enterprise project come later: whether the licence permits your use, how much GPU memory real traffic needs, what it costs against an API bill, and how you stop a prompt from drawing out what it should not. This guide is written for that decision. Our guide to LLM fine-tuning covers adapting an open model to your domain.
A self-hosted LLM is a large language model that an organisation runs on infrastructure it controls, such as its own servers, a private data centre or its own cloud account, instead of calling a model provider's API. Prompts, documents and answers stay inside that environment, and the organisation chooses the model, its version and who may use it.
In practice there are three pieces: an open model whose weights you can download, serving software that loads the model and answers requests, and hardware, usually GPUs, with enough memory to hold it. Applications talk to it much as they would talk to a commercial API. Many serving tools offer an interface compatible with OpenAI's, so existing code needs little change. You gain control and take on responsibility in equal measure.
The terms overlap. The useful difference is who controls the hardware and who can see the data.
A local LLM runs on the machine in front of you, such as a laptop or workstation. It is how most people first run an LLM locally, and it suits experiments, not a whole organisation.
A self-hosted LLM runs on infrastructure your organisation operates for many users: servers in your data centre, at the edge or in your own cloud account. An on-premise LLM is the same idea kept on your own premises. An air-gapped deployment goes one step further, with no connection to the internet at all.
A private LLM describes the outcome rather than the location: a model whose prompts and data are not shared with anyone else. A self-hosted model is private by design. A provider's dedicated cloud offering can also be private, but you then rely on its contract and controls, not your own.
Privacy usually opens the conversation. Cost and control often close it.
Data sovereignty and residency. India's Digital Personal Data Protection Act 2023 lets the government restrict transfers of personal data to countries it notifies, and stricter sector rules continue to apply alongside it. The EU's GDPR restricts transfers of personal data outside the European Economic Area unless adequate safeguards are in place. A model on your own servers removes the transfer question for every prompt, because the data never leaves.
Air-gapped and regulated sites. Defence, government, critical infrastructure, banking and healthcare often run networks with no route to the internet. For them, an API is simply unavailable.
Predictable cost at volume. APIs bill per token, so the bill rises with every user and every longer prompt. A self-hosted model costs roughly the same busy or idle. At high, steady volume that fixed shape can work out cheaper, and it is easier to budget.
Latency. A model in the same building as the application avoids internet round trips and shared queues, which matters for voice and for agents that call the model many times per task.
Control over model versions. Providers update and retire models on their own timetable. A self-hosted model changes only when you change it, which matters for anything you have validated or audited.
Five dimensions decide most comparisons. Neither option wins all five.
Privacy. Self-hosted: prompts and outputs stay on infrastructure you control. API: data goes to the provider under its terms and retention rules, which suits many uses but is never fully in your hands.
Cost shape. Self-hosted: mostly fixed, set by hardware, power, space and people, so the cost per request falls as use rises. API: fully variable, with nothing to buy upfront and a bill that grows in line with use.
Quality ceiling. Self-hosted: limited to open models that fit your hardware, which are strong and very capable on focused tasks. API: access to the largest frontier models, which are not available to download at all.
Operations burden. Self-hosted: you own capacity, upgrades, monitoring, patches and incidents. API: the provider runs the model, and your team integrates it.
Speed of change. Self-hosted: new models arrive when you choose to deploy them, which brings stability and some lag. API: new capabilities arrive the day they ship, along with changes you did not ask for.
Many organisations use both: a self-hosted model for sensitive, high volume work, and an API for occasional tasks that touch no regulated data.
Three things: a model you are allowed to use, software to serve it and hardware to run it. Choose them in that order.
The model, and its licence. Read the licence before the benchmark. Some open model families use permissive licences such as Apache 2.0. Others use custom community licences with conditions of their own. Meta's Llama 3.1 Community License, for example, requires any organisation whose products had more than 700 million monthly active users on its release date to request a separate licence from Meta. It also asks you to display "Built with Llama", and requires a distributed model built from it to carry "Llama" at the start of its name. None of this blocks most enterprises, but your legal team should read it before engineering commits to a model.
The serving layer. Serving software loads the model into GPU memory, batches requests and streams answers back. Widely used open tools include vLLM, built for high throughput serving on GPUs; Ollama, which makes it simple to run a model on a laptop or a single server; and llama.cpp, which runs models efficiently on CPUs and modest hardware. Hugging Face now lists its Text Generation Inference in maintenance mode and recommends engines such as vLLM and SGLang instead, a reminder to check a tool's status, not just its features.
The serving layer matters more than it looks. The 2023 paper that introduced vLLM, by Woosuk Kwon and colleagues, found that the memory each request holds is large and easily wasted, and that managing it better raised throughput by 2 to 4 times at the same latency.
The hardware. Usually GPUs, sized by memory first.
GPU memory is the binding constraint. The model must fit in memory to run at a useful speed, and so must the working memory of every conversation in flight.
Start with the weights. Hugging Face's guide to optimising LLMs for speed and memory gives a simple rule of thumb: a model needs roughly 2 GB of GPU memory for every billion parameters to load at 16-bit precision, the format most models are now trained in. A 7 billion parameter model needs about 14 GB. A 70 billion parameter model needs about 140 GB for its weights alone, which usually means splitting it across several GPUs.
Then add the conversations. While it answers, a model keeps a cache of the text it has already read, known as the key value cache. Hugging Face notes that it grows with the length of the conversation, and long documents and many users at once multiply it. Size for your real context lengths and peak number of users, not for one short prompt.
Quantisation, in plain words. Quantisation stores each of the model's numbers with fewer bits, much as a photo can be saved at lower quality to take less space. Hugging Face reports that weights can be reduced to 8 or 4 bits without a significant loss in performance. In its own worked example, a 15 billion parameter model needed around 30 GB at 16-bit precision, about 15 GB at 8 bits and under 10 GB at 4 bits.
The trade is real, though. The same guide warns that quantisation trades memory against accuracy and sometimes speed, so test the quantised model on your own tasks before you buy hardware on the strength of it.
The practical sequence: choose the smallest model that passes your evaluation, then the lowest precision that still passes, then size memory for the weights plus the peak cache, with headroom. Our comparison of small and large language models explains why the smallest model that does the job is so often the right one.
The best self-hosted LLM is the smallest model, under a licence you can accept, that passes an evaluation built from your own tasks. Leaderboards are a starting point, not an answer.
Public rankings measure general skills on public questions, while your workload is specific. Shortlist three or four models across two sizes, check the licence, languages, context length and serving support, then run the same test set through all of them and choose on the results.
A self-hosted LLM costs what its hardware, operations and people cost, spread across the requests it serves. Any figure given without your volumes is a guess.
The cost drivers are consistent: hardware, bought or rented and sized by memory; power, cooling and rack space; engineering time to build and integrate it; and ongoing operations, from monitoring to upgrades. Utilisation decides everything else, because an idle GPU costs the same as a busy one.
To find the break even point against an API, work from your own numbers.
1. Measure or forecast your monthly volume in tokens, with input and output counted separately, using real prompts that include any retrieved documents.
2. Price that volume at the current published rates of the API you would otherwise use. That is your monthly API cost.
3. Add up the monthly self-hosted cost: hardware spread over its useful life or the monthly rental, power and hosting, and the share of engineering and operations time it needs.
4. Load test the shortlisted model on the hardware you priced, to confirm it serves your peak demand at an acceptable speed. Then compare the two totals.
Pilot volumes rarely look like production, so model low, expected and high scenarios. And if an air gap makes an API impossible, the real comparison is between self-hosting and no model at all.
Self-hosting keeps your data away from a provider. It does nothing to stop the model itself from being misused. The OWASP Top 10 for LLM Applications, from the Open Worldwide Application Security Project, is the checklist to build against. Its 2025 edition starts with prompt injection.
Prompt injection. Text a user types, or text hidden in a document the model reads, can override your instructions. Treat everything the model reads as untrusted, and never let its output trigger an action without checks.
Sensitive information disclosure. A model can repeat data the person asking should not see. Filter documents by each user's permissions before the model sees them, and remove personal data the task does not need.
Excessive agency. OWASP warns that LLM systems are often given more autonomy than they need. Give a model the fewest tools and permissions the task requires, and ask a person to confirm high impact actions.
Supply chain and consumption. A downloaded model is software from outside your organisation, so take weights from the original publisher and record which version runs where. Set usage limits per user and per application, so a runaway loop cannot exhaust your GPUs.
Access control and logging. Put the model behind your existing identity system, so every request is tied to a person or a service. Log prompts, sources, outputs and tool calls, and protect those logs, because they hold the same sensitive data as the system. Our guide to big data security covers the wider controls.
An open model will not tell you when it is wrong. You need a way to measure it before launch and to notice when quality slips after.
Before launch, build a test set from real tasks, with answers your experts accept. Score accuracy, format, speed and safety under deliberate misuse. Run it on every candidate model, every quantised version and every upgrade.
After launch, watch three things. Service health: response times, errors and GPU memory. Quality: sampled answers reviewed by people, and the test set rerun on a schedule. Cost: tokens per team or application, so the value stays visible.
Keep the previous model ready to restore. A rollback that takes minutes is the cheapest safeguard you will build.
The path that works is narrow first and wide later.
1. Pick one use case with clear value, real users and data that justifies self-hosting.
2. Write the evaluation before choosing a model: real tasks, accepted answers and the bar for success.
3. Shortlist open models, licence first, and test them on rented or existing hardware before buying anything.
4. Choose the serving tool and the precision, then load test at realistic context lengths and numbers of users.
5. Build the security around it: identity, permission filtering, input and output checks, limits and logging.
6. Run a pilot with real users, measure against the bar and fix what fails.
7. Size production hardware from the pilot's measurements, with headroom and a standby.
8. Launch with monitoring, cost tracking and a rehearsed rollback.
Once the first use case is stable, the same platform can serve the next. That is where the economics improve.
Self-hosting is often the wrong choice. Choose an API, or wait, in these cases.
Low or uncertain volume. Idle GPUs are capacity you pay for and do not use. An API charges only for what you consume.
No team to run it. A self-hosted model needs people who can operate GPUs, patch software, respond to incidents and evaluate new models. Without them, the system decays.
You need the very largest frontier models. The most capable commercial models cannot be downloaded. If your task genuinely needs them, and your data rules allow it, use the API.
You are still exploring. Early prototypes move faster on an API. Self-host once the use case, volume and data rules are clear.
At Aptibit, we run language models on our clients' own infrastructure, so their data stays inside their network, even fully air-gapped. It is the same approach behind Radha, the AI copilot in our Visylix platform, which runs entirely on the customer's own servers with no cloud dependency. Radha works through 111 tools, asks for confirmation before high impact actions and keeps a full audit trail.
Every engagement starts with the use case, how success will be measured, and whether self-hosting is justified at all. We compare leading open models on fit for your domain, speed, cost and licensing. We test against your real tasks, not generic scores, including deliberate attempts to misuse the model, and tune the deployment to get the most from your hardware.
We launch with monitoring and records in place, and with safeguards that block misuse, protect personal data and keep the model on topic. A focused proof of concept takes 2 to 4 weeks. A full product typically takes 3 to 6 months. You get a detailed timeline before any work begins, and you see working software every two weeks. Code, models, data pipelines and documentation belong to you under an IP assignment signed before work begins, and all of it is senior engineering at India cost levels, well below US and Western European rates.
A self-hosted LLM earns its place wherever data must stay in your hands: regulated personal data, confidential documents, and air-gapped or sovereign sites. It also wins on high, steady volume, where a fixed cost beats a growing per token bill, on latency inside your own network, and wherever a validated model must not change without your say.
It disappoints at low or unpredictable volume, where idle GPUs cost more than an API, in organisations with no team to operate it, and on tasks that genuinely need the largest frontier models. It also disappoints when teams mistake privacy for security: a model on your own servers still needs permission filtering, defences against prompt injection, limits and logs.
A self-hosted LLM is a large language model that an organisation runs on infrastructure it controls, such as its own servers, data centre or cloud account, instead of calling a provider's API. Prompts, documents and answers stay inside that environment. It needs three things: an open model you are licensed to use, serving software that answers requests, and hardware, usually GPUs, with enough memory to hold the model.
A local LLM runs on a single machine, such as a laptop or workstation, usually for one person or for experiments. A self-hosted LLM runs on infrastructure an organisation operates for many users, with access control, monitoring and capacity for real traffic. An on-premise LLM is a self-hosted model kept on your own premises, and an air-gapped one has no internet connection at all.
Mostly GPU memory. Hugging Face gives a rule of thumb of roughly 2 GB per billion parameters to load a model at 16-bit precision, so a 7 billion parameter model needs about 14 GB for its weights. Add memory for the cache each conversation holds, which grows with context length and the number of users. Quantisation to 8 or 4 bits cuts the weight memory substantially.
Sometimes. An API bills per token, so cost rises with use. A self-hosted model costs about the same busy or idle: hardware, power, hosting and the people who run it. At high, steady volume self-hosting can cost less per request. At low or uncertain volume an API is usually cheaper. Compare both using your own token volumes and published API rates, not a generic figure.
The best self-hosted LLM is the smallest model, under a licence you can accept, that passes an evaluation built from your own tasks. Public leaderboards measure general skills on public questions, so use them only to build a shortlist. Test three or four candidates across two sizes on the same set of real tasks, then choose on accuracy, speed and the hardware each one needs.
Use the smallest model that does the job, quantise it to the lowest precision that still passes your tests, and run it on hardware you already own or rent by the hour while you prove the use case. Tools such as Ollama and llama.cpp run small models on a single server or even a CPU. For low or occasional volume, though, a pay per use API is often cheapest overall.
It helps, but it does not meet those obligations on its own. Running the model on your own servers means prompts containing personal data are not transferred to a model provider, which removes a major cross border question under India's DPDP Act 2023 and the EU GDPR. You still need a lawful purpose, permission controls, retention limits and records for the personal data the system processes.
Yes. Open model weights and the serving software can be brought into an isolated network and run with no internet connection. Updates, new models and security patches then arrive through your own controlled transfer process rather than a download. Air-gapped deployment is common in defence, government and critical infrastructure, where an API is simply not available.
Yes. Aptibit runs language models on clients' own infrastructure, including fully air-gapped networks, the same approach behind Radha, the on-premise AI copilot in our Visylix platform. We compare leading open models on fit, speed, cost and licensing, test against your real tasks and launch with monitoring and safeguards. A focused proof of concept takes 2 to 4 weeks, and you own the code and models.