For most enterprise AI projects, the default architecture is simple: send the request to a large model through a cloud API and pay per token. It is the fastest way to start, and for many tasks it remains the right choice. But for a growing share of production workloads, it is also the most expensive, the slowest and the hardest to defend to a privacy officer.
The alternative is no longer a research project. Small models that run on a single server inside your own environment can now handle much of the routine work that enterprise AI actually does. The question for technology leaders is not whether small models are good enough in general. It is which of your workloads they are good enough for.
What counts as a small model
There is no official cut-off, but a useful working definition comes from NVIDIA researchers, who in their paper Small Language Models are the Future of Agentic AI treat most models below 10 billion parameters as small. In practice, these are open-weight models you can download, fine-tune and run on your own hardware or private cloud, rather than rent through someone else’s API.
Why the question is live now
Two trends have changed the economics. First, small models have become far more capable. Stanford’s AI Index 2025 reports that, driven by increasingly capable small models, the inference cost for a system performing at the level of GPT-3.5 dropped more than 280-fold between November 2022 and October 2024.
Second, enterprises are moving toward specialized models. Gartner predicts that by 2027 organizations will use small, task-specific AI models at least three times more than general-purpose large language models, noting that they provide quicker responses and use less computational power.
Five signs a small private model is the better fit
1. The data cannot leave your environment
Health records, client financial data, legal files and HR cases often carry rules about where they can be processed. The Government of Canada’s guide on the use of generative AI is blunt about it: public servants must not enter personal information into publicly available generative AI tools, because the supplier might store a copy. Private-sector teams in healthcare, insurance and finance face similar questions from their own privacy and compliance leads. A model running inside your network removes the third party from the data flow entirely.
2. The task is narrow and repeatable
Classifying tickets, extracting fields from documents, routing requests, redacting personal data and summarizing to a fixed template do not need a model that can write poetry and debug code. A small model fine-tuned on a few thousand of your own examples often matches a much larger general model on that one task.
3. Volume is high and steady
Per-token pricing scales in a straight line with use. Owned or reserved infrastructure is closer to a fixed cost. The NVIDIA paper estimates that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and compute than serving a 70 to 175 billion parameter model. At millions of requests a month, that gap decides the business case.
4. Speed or connectivity matters
A model on the same network as the application avoids an internet round trip and a shared queue. That matters for anything a person is waiting on in real time, and for sites where connectivity is limited or must be cut off, such as plants, clinics, vessels and field operations.
5. You need a model that does not change underneath you
Cloud providers update and retire models on their schedule. For regulated decisions, you may need to show exactly which model produced an output and get the same behaviour next quarter. With your own weights, the version only changes when you change it.
When the cloud model still wins
Small models are not the answer to everything. A large hosted model is usually the better choice when:
- The task is open-ended, needing broad world knowledge, long multi-step reasoning or many kinds of request in one place.
- Volume is low or unpredictable, so idle hardware would cost more than the API bill.
- You are still exploring, and the use case may change next month.
- Nobody owns the platform. Self-hosting means GPUs, patching, monitoring, scaling and security. If no team runs that, the savings disappear quickly.
The pattern most teams land on: route, don’t choose
The strongest designs rarely pick one side. A small private model handles the sensitive and routine requests, and only the hard cases go to a large model, after personal data has been removed. The NVIDIA researchers describe the same idea: systems that use different models for different calls, with a large model kept for the steps that truly need it. The routing layer then needs the same controls as any other AI system, from input checks to logging, which we cover in our guide to AI agent guardrails.
Decide with numbers in four weeks
Opinions about model size are cheap. A short proof of concept settles the question for one workload:
- Pick one workload with real volume and a clear owner.
- Build an evaluation set of a few hundred real, labelled examples.
- Run a cloud baseline and two or three small open-weight models, with and without light fine-tuning.
- Measure what matters: accuracy on your examples, response time at the slow end (not just the average), cost per thousand requests at projected volume, and where the data goes.
If a small model comes within your accuracy tolerance at a fraction of the cost, you have a business case. If it does not, you have evidence for staying on the API, which is just as useful.
Build it with 247 Labs
247 Labs designs and builds AI systems for organizations that need control over their data, from model evaluation and fine-tuning to the infrastructure and operations that keep a private model running. If you are weighing a cloud API against a model of your own, talk to our team about testing it on one real workload.
