Tokenization is how an AI model chops text into pieces called tokens before processing it. A token can be a whole short word, part of a longer word, or a piece of punctuation. The model never sees raw text. It only sees tokens. Every AI bill is calculated in tokens. Every context window is measured in tokens. Tokenization is invisible but it is the unit your AI spending is denominated in.
Key Takeaways
- A token is a small chunk of text that an AI model treats as a single unit. The model sees tokens, not words or characters.
- For English, the rough rule is one token equals about 4 characters, or about 0.75 words. A 1,000-word prompt is roughly 1,300 tokens.
- Almost every AI vendor (OpenAI, Anthropic, Google) bills per token. Input tokens and output tokens are usually priced differently. Understanding token costs is how you predict and control AI spending.
- Different models use different tokenizers. Same content can use slightly different token counts on GPT-4o vs Claude vs Gemini, but rarely by more than 10-15%.
- Non-English languages can be 2-3x more token-expensive than English, which matters for any SMB running AI features for non-English-speaking customers.
In Simple Terms
A large language model cannot read text directly. Before any GPT, Claude, or Gemini call runs, the input text gets broken into tokens, which are small pieces that the model has learned during training. Common short words become single tokens. Longer or rarer words get split into multiple tokens. Spaces, punctuation, and special characters each get their own treatment.
For an SMB, the practical importance of tokenization is cost. You are not billed by word, sentence, or document. You are billed by token. Knowing roughly how many tokens your typical prompts and responses use is the difference between predictable AI spending and a billing shock.
The shortcut most people learn is: in English, one token is about four characters, or about 0.75 of a word. A 500-word email is around 650 tokens. A 10,000-word document is around 13,000. Those numbers are close enough to budget on. Exact counts come from a tokenizer (OpenAI's free tiktoken library and Anthropic's documentation both let you see exact counts).
How Tokenization Affects Cost
Three practical points that matter when budgeting AI use.
Input and output tokens are priced separately. Output usually costs 3 to 5 times more per token than input. The model is generating those tokens one at a time using compute. GPT-4o in early 2026 prices around $2.50 per million input tokens and $10 per million output tokens. Claude Sonnet 4 sits at $3 input and $15 output. Doubling the length of a response doubles the output cost, while a longer prompt only modestly increases the input cost.
Conversation history adds up. A back-and-forth chat resends the whole conversation as input on each new turn. By turn ten of a long conversation, you might be sending 5,000 tokens of context just to get a 100-token reply. This is one of the biggest sources of unexpected AI bills in SMB use. Tools that summarise older history before resending it (or RAG-style retrieval) keep costs under control.
Languages other than English are more expensive. The tokenizers were optimised on English-heavy training data. A French paragraph tokenizes slightly worse than the same paragraph in English. Korean, Japanese, Mandarin, and Hindi can come in at 2 to 3 times more tokens for the same actual content. For an SMB running an AI feature in multiple languages, this is worth measuring before scaling. The user experience may be identical; the cost per query is not.
Tokenizer Variations Between Models
Different models use different tokenizers, each with its own vocabulary.
OpenAI's tokenizers use Byte-Pair Encoding (BPE). The current GPT-4o tokenizer has a vocabulary of about 200,000 tokens, larger than its predecessor (the cl100k_base tokenizer used by GPT-4 and GPT-3.5 had about 100,000). A larger vocabulary means each common phrase fits into fewer tokens.
Anthropic's Claude uses a different tokenizer, also subword-based. Token counts on the same content typically differ from OpenAI's by less than 10%. Anthropic publishes a counting endpoint so you can check exact token usage before sending requests.
Google's Gemini uses its own tokenizer optimised for the 1M-2M context window. Gemini tokenization is typically more efficient on long documents but less optimised on short conversational text.
For SMB use, the differences rarely matter at the per-call level. They matter when comparing total cost between providers at high volume. The cleanest way to compare is to take a sample of your real requests, count tokens in each provider's tokenizer, and price the same workload across all three.
Frequently Asked Questions
What is a token, in practical terms?
Why do tokens matter for my AI bill?
Are all tokens the same across models?
Are non-English languages tokenized differently?
What is a context window, and how does it relate to tokens?
Related Glossary Terms & Resources
Foundation Model
The category of model that takes tokens as input. Tokenization is the first step in everything they do.
Inference
What models do with the tokens once they are tokenized. Inference cost scales with the number of tokens.
Embedding
Embeddings are produced from tokenized text. Embedding models have their own tokenization rules.
API
Where token counts show up in practice: in API responses and on usage bills from OpenAI, Anthropic, and others.