Token Counter

What is a token?

A token is the unit of text a language model reads and writes. A common word is usually one token, together with the space in front of it. A rarer or longer word is split into pieces, and punctuation, numbers and code symbols often get tokens of their own. OpenAI, Anthropic and Google all bill API use by the token, so the count decides what a prompt costs and whether it fits in a model's context window.

Example: “Unbelievably, the tokenizer handled it.” has 5 words and 39 characters. GPT-5's o200k_base encoding turns it into 9 tokens: Un bel ievably , ␣the ␣tokenizer ␣handled ␣it . (␣ marks a space). Each common word is one token, while “Unbelievably” becomes three.

Encodings differ, too. “I can't believe it's already November!” is 7 tokens on GPT-5 but 9 on GPT-4, because GPT-4's older cl100k_base encoding splits “can't” and “it's” into two tokens each. Which encoding a model uses comes from OpenAI's tiktoken library: o200k_base for GPT-5, GPT-4.1, GPT-4o, o1, o3 and o4-mini, and cl100k_base for GPT-4 and GPT-3.5 Turbo.

How many tokens is a word?

For English, OpenAI's rule of thumb is that 1 token is about 4 characters or 0.75 words, which is about 1.33 tokens per word. Google gives a similar rule for Gemini: a token is about 4 characters, and 100 tokens are about 60 to 80 English words. The sample prompt in the counter above has 68 words and 373 characters and comes to 81 tokens on GPT-5, or 1.19 tokens per word and 4.6 characters per token.

Code, numbers, emoji and most languages other than English need more tokens for the same amount of meaning. The table shows the same sentence in four languages and a few other kinds of text, counted with both OpenAI encodings.

The Hindi sentence takes 27 tokens on GPT-5 but 83 on GPT-4, so the newer encoding is much cheaper for some languages. For English text and code the two encodings give nearly the same count. To turn a token count into dollars on Claude, GPT or Gemini models, use the LLM API cost calculator.

How to count Claude and Gemini tokens

Claude and Gemini models use their own tokenizers, so the same text gives a different count on them than on GPT-5. An OpenAI tokenizer cannot tell you a Claude or Gemini count, which is why this counter labels its numbers with the OpenAI models they belong to.

For Claude, Anthropic offers a free token counting endpoint, POST /v1/messages/count_tokens, that takes the same input as a normal message (system prompt, tools, images, PDFs) and returns the input token count for the model you name. It needs an API key and has its own rate limit, starting at 5,000 requests per minute. Anthropic calls the result an estimate that can differ from the actual input by a small amount. It also notes that Claude Opus 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text than earlier Claude models, so count against the model you plan to use.

For Gemini, Google's API has a countTokens method (count_tokens in Python) that returns the input token total before you send a request, and every response reports the tokens it used. The Gemini row in the counter is only a rough estimate based on Google's rule of about 4 characters per token.

Questions