A token is the unit that models read and the unit your provider charges for: roughly four characters of English, or three-quarters of a word. You’re not charged per question or per answer, but per token, in both directions and on every request.
That distinction is why a bill can surprise you. Two questions of the same length can cost vastly different amounts, and a conversation gets more expensive as it goes on even if your questions stay short.
By Devy · Category: AI Dev Tools
What is a token in AI?
A token is the minimum unit in which a model processes text. Anthropic’s glossary defines them as “the individual smallest units of a language model, and can correspond to words, subwords, characters, or even bytes”. OpenAI puts it more directly: “A token can represent a character, part of a word, a whole word, or a punctuation mark. Spaces also affect how text is split into tokens”.
The model never sees your text as text. Before anything happens, your input is encoded into a sequence of tokens, the model processes that sequence, and its output is decoded back. Tokens are the native unit; characters are the interface.
Why do they charge per token and not per question?
Because the token is what the model actually processes, and the cost scales with how many there are.
The practical consequence is something almost everyone misses: you pay for the entire conversation on every turn. Models are stateless. To answer your tenth message, the model receives the nine previous exchanges plus the system prompt plus any attachments, all resent as input. Your tenth question might be six words, and the request carrying it twenty thousand tokens.
That’s why an agent working on a long task costs more per step as it goes on, and why “the conversation got more expensive and I didn’t do anything different” isn’t a billing error.
Input and output are charged at different prices—output usually costs several times more per token—so a short prompt asking for a long document has a very different economic shape than a long prompt asking for a yes or no.
How many tokens does your text have?
The published rules of thumb, for English:
| Source | Rule |
|---|---|
| OpenAI | ~4 characters per token; ~¾ of a word; 100 tokens ≈ 75 words |
| Anthropic | ~3.5 characters of English per token |
OpenAI is explicit that these are approximations: “These are estimates, not exact counts. The lengths of sentences and paragraphs vary, and other languages may have different relationships between characters, words, and tokens”.
Notice that the two numbers don’t match—4 versus 3.5—and neither is wrong. Each model family has its own tokenizer. The same text doesn’t have a single token count.
If you want the token count for your own text without writing code, the yoDEV token calculator counts it and estimates the cost for current OpenAI and Anthropic models. The count happens in your browser: what you paste doesn’t leave your machine.
What is a tokenizer and why does it change the count?
The tokenizer is the component that decides where your text gets cut, and each model family uses its own.
Its logic is economic. Anthropic explains the tradeoff bluntly: “Larger tokens allow data efficiency during inference and pretraining (and are used when possible), while smaller tokens allow the model to handle rare or unseen words”. That is: divide frequent sequences into large chunks and rare ones into small pieces.
From that comes the rule most worth taking to heart. A common word is usually one token. An unusual proper name, a long identifier from your code, or a base64 blob can be many. Predictable text is cheap; rare text is expensive.
Does Spanish use more tokens than English?
Less than you’d imagine, and it’s worth correcting this idea.
There is a real phenomenon where languages poorly served by a tokenizer get split into many more tokens to say the same thing, and pay more for it. But the per-language table OpenAI published places Spanish at the efficient end. When they introduced the GPT-4o family tokenizer, Spanish went from 29 tokens to 26 on their sample sentence: an improvement of 1.1×. Gujarati improved 4.4× and Telugu 3.5×.
It’s worth reading that straight. A small improvement means Spanish had little to gain, because it was already tokenizing reasonably well. The languages that gained a lot are the ones that were being penalized severely. Spanish shares an alphabet and much of its subword structure with English, so it was never in that category.
What does cost you extra is anything the tokenizer finds unusual: code identifiers, technical strings, and text that mixes languages mid-sentence.
Why does the same text cost more than before?
Because tokenizers change between model generations, and yours might have changed without you noticing.
Anthhropic’s documentation describes the tokenizer that debuted with Claude Opus 4.7 as one that “produces approximately 30 percent more tokens than earlier models for the same input text”. That same tokenizer is shared by Opus 4.8 and the Fable 5 family. The actual increase depends on the text: it’s a range that reaches about 1.35 times the previous count, not a fixed figure.
It’s worth reading the scope carefully, because “4.7 onwards” doesn’t equal “all recent models”. Sonnet 5 and Haiku 4.5 still use the earlier tokenizer. If your workload runs on either of those two, there’s nothing to recalculate.
If you budgeted a workload for Opus 4.6 or an earlier model and later migrated to Opus 4.7, Opus 4.8, or Fable 5, your estimate can come up short by as much as a third before any other variable even enters. Re-measure after any model change. This is exactly the kind of fact worth checking against your provider’s documentation at planning time, instead of dragging a number from last year.
How do you count tokens before paying for them?
Anthhropic exposes a token counting endpoint and using it is free, subject only to request limits per minute: from 5,000 per minute at the Start tier up to 20,000 at Scale. It returns the count and nothing else:
{ "input_tokens": 14 }
It counts the actual request, not an approximation: messages with tools attached return a higher figure—their documentation example returns 403—and messages with an image, even more: 1,028 in their example. Their documentation also clarifies that “you are not charged for the tokens added by the system. Billing reflects only your content”.
If you’re sizing a workload, measuring beats estimating. A rule of thumb has no idea what your prompts contain.
What is the context window?
The context window is the ceiling on how many tokens the model can consider at once. Anthropic describes it as “a ‘working memory’ for the model”, distinct from the corpus it was trained on.
Cost and context are two different constraints on the same number. The window tells you when a request will fail; the price tells you how much it costs when it works. A long conversation runs into the second problem long before the first.
Where does the spending actually go?
Almost never where you look first. The prompt you wrote is rarely the expensive part.
The expensive parts are the ones you didn’t write: the conversation history resent on every turn, the system prompt and tool definitions attached to each request, the files an agent loaded into context, the retries. At yoDEV we cover each one separately:
- Headroom: how to cut up to 95% of your tokens without changing responses
- Your 5-minute cache doesn’t last 5 minutes: what promptCacheTtl actually fixes
- Cursor now automatically routes every request
- 3.5× fewer tokens and the same score: Toast 1 handles your entire agent search loop
- Agentmemory: persistent memory for Claude Code — 92% fewer tokens
- How to use Headroom with LiteLLM to spend fewer tokens — the case of the router charging you two calls per request
They all come down to the same arithmetic. Measure first, using the free endpoint, on your actual traffic. Then decide which of these levers is worth pulling.