Inference is what happens when a trained AI model gets used. You type a question into ChatGPT, the model generates an answer. That is inference. A churn-prediction model scores a customer record. That is inference. A camera classifies an image as cat or dog. That is inference. Inference is the production-phase work of AI: the part that runs every time the model is asked to do something useful, as opposed to training, which happens once.
Key Takeaways
- Inference is using a trained model. Training is teaching it. Two different jobs, two different cost profiles.
- Inference is the part of AI that costs ongoing money. For every dollar a company spends training a model, it can spend 15 to 20 dollars running inference on that model over its lifetime.
- The cost per inference has dropped about 50-fold since late 2022. GPT-4-class capability now costs about $0.40 per million tokens, down from around $20 per million tokens at GPT-4's launch.
- For SMBs, inference is most often invisible: it is the cost the AI vendor (OpenAI, Anthropic, Google, HubSpot AI, etc) absorbs and recovers through subscription or API pricing.
- When building custom AI agents, choosing the right model size for the job is where most inference-cost overruns get fixed. A smaller cheaper model usually does most work fine.
In Simple Terms
Training an AI model is like teaching someone a skill. It takes a long time, costs a lot of money up front, and happens once. Inference is what the trained person does for a living afterwards: answering questions, making predictions, generating content. Every interaction with the model is an inference. Every output is the result of inference running.
For an SMB owner, inference is the part of AI you actually pay for over time. Training GPT-4 cost OpenAI an estimated $100 million-plus. You will never pay anything close to that. But every time someone in your business uses ChatGPT, an inference call runs, and the cost of those calls is what your subscription is covering.
Inference is also where most of the recent cost improvements have happened. The same task that cost $20 in late 2022 costs about $0.40 in early 2026, a roughly 50-fold drop in three years. That is why AI features have become economically reasonable to embed in everyday SMB tools (HubSpot AI, Microsoft Copilot, Notion AI, Intercom Fin) when they were not in 2022.
Why Inference Costs Have Fallen So Fast
Three drivers operating at once.
Better hardware. NVIDIA's H100 and Blackwell GPUs, Google's TPU v5 and Trillium, AMD's MI300 and MI350, and specialised inference chips like Groq's LPUs have each cut the cost-per-token of running a model. NVIDIA H100 cloud pricing alone has fallen from $8-10 per GPU-hour at peak to under $3 per hour by early 2026.
Better software. Model distillation produces smaller versions of large models that keep most of the capability at a fraction of the inference cost. Quantisation reduces the precision of model weights with little accuracy loss. Speculative decoding predicts the next few tokens before the model finishes the current one. Each technique trims compute per inference by a meaningful percentage; combined they compound.
Pure competition. OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, and a half-dozen others are all selling inference. DeepSeek's late-2024 entry at roughly 90% lower pricing than incumbents triggered another round of frontier-model price cuts. The dynamics look like the AWS-vs-Azure cloud wars of the early 2010s, just on a faster clock.
What This Means for SMBs Choosing AI Tools
For most SMB cases, you do not see inference costs directly. You see a monthly subscription or an API bill, and the vendor handles inference under the hood. The practical things to watch:
Model choice in custom builds. If you are commissioning a custom AI agent or using a tool that lets you pick the model (n8n, Make, custom apps via API), the difference between using GPT-4o ($2.50 per million input tokens, roughly) and GPT-4o-mini ($0.15) is 16x on cost. For most internal automation, the smaller model is fine. Pay the premium only where the task genuinely needs the frontier model's reasoning.
Customer-facing volume. An internal team using ChatGPT 50 times a day is a fixed-price problem. A customer-facing chatbot answering thousands of conversations a day is an inference-cost problem. Build the cost model before you launch a high-volume AI feature.
Latency vs capability. The more capable the model, the slower the inference. Real-time use cases (voice agents, instant chat) often need smaller models or specialised inference hardware. Background use cases (overnight email triage, weekly report generation) can use the slowest, most capable model without anyone noticing.
The trend is your friend. Whatever you build today will be cheaper to run tomorrow. The inference cost curve has consistently surprised even AI insiders on the downside. Designing AI features that assume costs will keep falling is a safer assumption than designing on today's prices.
Frequently Asked Questions
What is the difference between training and inference?
Why has inference become so much cheaper?
What does inference cost an SMB in practice?
Why do AI agents seem slow sometimes?
What should an SMB watch for on inference costs?
Related Glossary Terms & Resources
Machine Learning
The category. Inference is the using-it half of machine learning, after training.
Foundation Model
The category of large pre-trained models that businesses now use for inference via API.
API
How most SMBs access inference: as a paid API call to OpenAI, Anthropic, Google, or others.
AI Automation Statistics 2026
Market and cost data across AI inference, training, and deployment.