Token counts, not words, decide what a prompt costs

A fix for words a translation model had never seen

In a paper submitted on 31 August 2015, Sennrich, Haddow and Birch proposed encoding rare and unknown words as sequences of subword units rather than treating every word as an indivisible symbol. Their method adapted byte pair encoding, originally a compression algorithm, to build a fixed vocabulary of frequently occurring character sequences: common words stay whole, rarer words split into recognisable fragments, and a model never needs an out-of-vocabulary symbol for a name or compound it has not seen during training. The paper reported improvements of 1.1 to 1.3 BLEU points over dictionary-based baselines on English-German and English-Russian translation, a modest number from a mechanism that later became close to universal in language models built for reasons that have little to do with translation.

What a modern tokenizer does with that idea

OpenAI's tiktoken library, documented as retrieved on 16 September 2026, applies the same subword logic through byte-pair merges and states that the conversion is reversible and lossless: tokens can always be turned back into the original text, and the scheme works on arbitrary input, including text unseen during training. The library's own description puts the practical rule of thumb at about four bytes of text per token, and it ships several encodings, including cl100k_base and o200k_base, tied to different model families rather than one encoding for every model. OpenAI's tokenizer page, also a living document as retrieved on 16 September 2026, states that a token typically corresponds to about four characters of common English text, or roughly three-quarters of a word.

Why the unit you are billed in is not the unit you read in

A model's context limit and a hosted API's price are both denominated in tokens, not words or characters, and the roughly-four-characters rule is an average over the English text these tools were tuned on. Unusual formatting, code, or a language with different morphology can tokenize far less efficiently, especially a script poorly represented in the merge vocabulary. Neither cited document promises a fixed ratio for any specific input, and a count run on one tokenizer does not transfer to a model using a different encoding, so a word-count-based estimate can be wrong before a single token is generated.

Questions to carry into your own evaluation

• Which encoding does the model you are billed against actually use, and does that match the tokenizer you tested with?
• How does your token count change for the languages or formats your real inputs use, compared with plain English prose?
• Does a prompt template, system message, or retrieved passage change the token count in a way a per-word estimate would miss?

The subword mechanism proposed for translation in 2015 is now the layer that determines what a request costs and how much of a context window it consumes, which makes counting tokens directly, on the model's own tokenizer, a more reliable habit than estimating from word counts.