“Every word you type to an AI is quietly broken down into fragments before it’s processed. Understanding tokens is the key to understanding how modern AI models actually work — and why they cost what they do.”
The building blocks of language
When you send a message to an AI like Claude or ChatGPT, it doesn’t read your words the way you do. Instead, instead they first convert your text into tokens, small chunks of text that form the model’s basic unit of understanding. This token may be a whole word, part of a word, a punctuation mark, or even a single character.
You can think of it like this: a newspaper printing press doesn’t deal in stories — it deals in individual letters and blocks of type. Tokens are the AI equivalent of those individual blocks.
HOW A SENTENCE GETS TOKENISED
11 tokens — notice “jumped” splits into “jump” + “ed”, a common sub-word pattern.
English text typically averages around ¾ of a token per character, or roughly 100 tokens per 75 words. A typical paragraph like this one contains somewhere between 50 and 100 tokens.
~75
words per 100 tokens
~4
average chars per token
1:1.3
words to tokens ratio
Why not just use words?
The English language contains hundreds of thousands of words, and that’s before you factor in slang, brand names, technical terms, and other languages. Teaching an AI to recognise every single one would be an enormous task, and it still wouldn’t cover words that get invented tomorrow. Although they are certainly getting better!
Instead, AI models break language into smaller, reusable pieces. This means that even if a model has never for instance seen the word “tokenisation” before, it can still make sense of it. This is because it recognises “token”, “isa”, and “tion” as familiar building blocks. It’s a bit like a child sounding out an unfamiliar word using the syllables they already know.
This keeps the model’s vocabulary to a manageable size – typically around 50,000 to 100,000 chunks. At the same time it is still able to handle almost anything you throw at it.
“Tokenisation is the unglamorous plumbing of AI. Invisible when it works, and the source of strange failures when it doesn’t.”
Tokens in practice
Scenario 1 — Drafting a LinkedIn post
You ask an AI to write a 200-word LinkedIn post about a product launch. You provide a brief with bullet points (~150 words) and receive a polished post in return (~200 words).
~200 tokens in (your prompt)~270 tokens out (the post)Total: ~470 tokens
Scenario 2 — Summarising a long document
You paste in a 10-page business report (~5,000 words) and ask for a concise executive summary. The report is nearly all input; the summary is relatively short output.
~6,700 tokens in~400 tokens outTotal: ~7,100 tokens
Scenario 3 — Writing and debugging code
A developer pastes 300 lines of code, describes a bug, and receives a corrected version with an explanation. Code tends to tokenise densely — variable names, punctuation, and indentation all count.
~1,200 tokens in~1,400 tokens outTotal: ~2,600 tokens
Tokens and cost
For anyone using the AI APIs — building apps, automating workflows, or running bulk analysis — tokens are directly tied to cost. API providers charge per million tokens, split between input and output (output is typically more expensive, as generating text is more computationally intensive than reading it).
This is why prompt engineering matters: a well-crafted, concise prompt uses fewer tokens and gets better results. Verbose system instructions, pasting in unnecessary context, or asking for excessively long outputs all add up quickly at scale.
Context windows: the token budget
Every AI model has a context window, that is the maximum number of tokens it can hold in “working memory” at once. This includes the full conversation history, any documents you’ve shared, and the response being generated. Modern models like Claude have context windows of 200,000 tokens or more, equivalent to a short novel.
When a conversation exceeds the context window, the model can no longer “see” the earliest parts of the chat. Because of this you will find very long conversations can sometimes feel like the AI has forgotten what was discussed at the start.
“Tokens may be invisible to most users, but they underpin everything, from how AI models comprehend language to how much a deployment costs at scale. Next time you’re crafting a prompt, remember: every word is quietly being counted.“
