What Are Tokens in AI? A Practical Explanation

A token is the unit of text a language model actually reads and writes. It is not a word and not a character: depending on the language and context, a token can be as short as a single character or as long as a full word. Everything you send to a model — and everything it sends back — is counted in tokens, which is why token counts determine context limits, processing load, and API billing.

This guide is for technically curious beginners and intermediate practitioners who want a working mental model of tokens rather than a linguistics lecture. It covers how tokenization works, how to estimate and measure counts, where those counts show up in real projects, and the mistakes that lead to blown context windows and unexpected costs.

What a token is, and why models need them

Language models do not process text directly. Tokenization breaks text into smaller units — words, parts of words, or individual characters — and the model works with numerical representations of those units to learn patterns and generate language. Stanford HAI’s plain-language definition states it the same way: tokenization is the process of breaking down text into smaller units called tokens.

The set of unique tokens a model was trained on is its vocabulary. Microsoft’s .NET documentation describes the pipeline: tokenization runs as the first step during training, the model analyzes semantic relationships between tokens — such as how commonly they appear together — and at inference time it generates a sequence of output tokens based on the input sequence.

GPT-family models use a specific method called Byte-Pair Encoding (BPE), a form of subword tokenization. The practical consequence is that common words tend to become a single token, while rare words, typos, unusual casing, and unfamiliar syntax get split into several tokens.

An illustrative split

Microsoft’s documentation uses the sentence I heard a dog bark loudly at a cat and shows it tokenized as I, heard, a, dog, bark, loudly, at, a, cat. That is a teaching example of word-level tokenization, not a transcript of what a BPE tokenizer produces. Real tokenizers merge frequent character pairs and split less common ones, so the actual output for that sentence differs by model and encoding.

Why the same text produces different token counts

OpenAI’s help center states plainly that tokenization varies by model and encoding, and that spaces, punctuation, and partial words all contribute to the count. Two consequences follow for anyone building with these tools:

• Counts are not portable. A prompt measured against one model’s encoding is only an estimate for another model’s.
• Character length is a poor proxy. A dense block of code, identifiers, or JSON can consume far more tokens than the same number of characters of ordinary English prose.

Microsoft’s tradeoff table explains the underlying tension. Smaller tokens — character or subword units — let a model handle a wider range of inputs such as unknown words, typos, or complex syntax, and can keep the vocabulary smaller, which requires fewer memory resources. The cost is that a given text is broken into more tokens, requiring additional computational resources, and a fixed token limit covers less text.

Estimating token counts without a tokenizer

For English prose, OpenAI publishes rules of thumb that are accurate enough for planning:

These ratios break down outside English prose. Use them to sanity-check a budget, not to bill a client or to size a context window you intend to fill to the brim.

Measuring tokens exactly

When the number matters, measure it. OpenAI recommends its interactive Tokenizer tool, which shows how text is broken into tokens, or the tiktoken library programmatically — a fast BPE tokenizer built for OpenAI models — using tiktoken.encoding_for_model(model) to get the exact count for a target model.

The following snippet is an illustrative, untested example of the documented pattern. It is not a benchmark result and its output is not reproduced here.

import tiktoken

# Use the model name you actually plan to call.
enc = tiktoken.encoding_for_model("gpt-4o")

text = "You miss 100% of the shots you don't take"
tokens = enc.encode(text)

print(len(tokens))   # token count
print(tokens)        # token IDs, for inspection

OpenAI’s documentation lists that Gretzky quote as 11 tokens. Run the snippet against your own model’s encoding and compare — the gap between the documented figure and your result is a useful reminder of how encoding-specific counts are. For local models, use the tokenizer that ships with the runtime or is referenced on the model card rather than an OpenAI encoding, because the vocabulary is different.

Where token counts show up in practice

Context windows and output limits

The context window is the total number of tokens a model can consider in a single request, covering both the input you send and the output it generates. A long document, a chat history, and a system prompt all draw from the same budget. When you exceed it, requests are truncated or rejected depending on the stack you are using.

Billing categories

OpenAI tracks usage in several categories, and the distinction matters when you are optimizing cost:

• Input tokens — tokens in your request.
• Output tokens — tokens generated in the response.
• Cached tokens — reused tokens in conversation history, often billed at a reduced rate.
• Reasoning tokens — in some advanced models, extra internal thinking steps included before the final output.

These counts appear in API response metadata and are used for billing and usage tracking. Logging that metadata is the cheapest way to learn where a workflow’s tokens actually go.

Calibration figures

OpenAI’s documentation offers three real-world samples for calibrating your intuition: the Gretzky quote at 11 tokens, the OpenAI Charter at 476 tokens, and the US Declaration of Independence at 1,695 tokens. A policy document or long-form article you plan to feed into a pipeline is likely in the same order of magnitude as that last figure.

Illustrative scenario: sizing a content pipeline

Suppose a workflow sends a 1,500-word style guide plus a 1,500-word draft on every revision pass. Using OpenAI’s English ratios, each block is roughly 2,048 tokens, so one pass carries about 4,096 input tokens before any instructions or conversation history are added. Three revision passes over the same draft therefore send on the order of 12,000 input tokens of largely repeated material. This is an estimate derived from published ratios, not a measurement — but it is exactly the arithmetic that explains why caching and summarization change a pipeline’s cost profile. Measure with the tokenizer before committing to a design.

Budgeting tokens in local and automated workflows

Local models

When you run a model locally, the context length is a hard constraint you configure, and it affects memory use beyond the model weights themselves. Longer contexts require additional memory for cached attention state and runtime overhead, so “the weights fit” is not the same as “this context length fits.” Measure peak memory at the context length you intend to use rather than extrapolating from the model file size.

WordPress and automation pipelines

Content workflows are where token counts quietly accumulate. A typical generative step — fetch source material, summarize, draft, revise, publish — can send the same background context several times. Habits that keep this predictable:

1. Count the tokens of each fixed block (system prompt, style guide, reference material) once, then treat it as a per-call floor.
2. Chunk long documents before sending them, and count each chunk rather than estimating from word count.
3. Pass summaries forward instead of full transcripts when the downstream step only needs the gist.
4. Read the usage metadata from each response and log it per step, so you can see which stage dominates.
5. Set a token budget per step and surface an error when a step exceeds it, rather than silently truncating.

Character limits in WordPress fields, database columns, or plugin settings do not map cleanly to tokens. If a plugin or API enforces a token limit, measure with the tokenizer for the model in use; if it enforces a character limit, that is a different constraint entirely.

Common mistakes and their fixes

• Treating one token as one word. The English rule of thumb is roughly ¾ of a word per token, so a 1,000-word article is closer to 1,300 tokens than 1,000.
• Reusing a count across models. Encodings differ; re-measure whenever you switch models or providers.
• Estimating from characters for non-prose input. Code, JSON, URLs, and identifiers tokenize less efficiently than English prose.
• Ignoring output and reasoning tokens. Input-only estimates understate the total, especially with models that emit internal reasoning tokens.
• Assuming cached tokens are free. They are often billed at a reduced rate, not at zero.

What to do next

Pick the model you actually plan to use, tokenize a representative sample of your own text with the matching tokenizer, and record the result. Compare it against the rules of thumb above. That single measurement tells you more about your real context and cost budget than any general ratio, and it gives you a baseline to re-check whenever you change models, prompts, or chunk sizes.

If a measured count surprises you, share the model, the tokenizer, and a redacted text sample in the community so others can reproduce it.