Why Does ChatML Add a 7-Token Overhead per Message?

Below is an explanation of the extra tokens added in the sample code. In ChatML, every message isn’t encoded as a simple, literal split of the visible characters. Instead, the text is first wrapped in a hidden formatting that tells the model where a message begins, what role that message has, and where it ends. The numbers (4 and 3) come from how the model’s internal tokenizer encodes these hidden formatting markers, not from simply counting characters or visible words.

The 4 Tokens per Message

The sample code assumes that every message is internally formatted as something like:

  <|start|>{role/name}\n{content}<|end|>\n

When this string is processed by OpenAI’s tokenizer (using, for example, the tiktoken library), the hidden markers and newline characters are grouped into four tokens. Although it might appear that there are more “pieces” in that string, the tokenizer groups certain parts together into a single token, so that the entire format overhead counts as 4 tokens per message.

The 3 Tokens for the Reply

Similarly, when the API prepares a reply it uses an internal prompt that starts like:

  <|start|>assistant<|message|>

Again, when tokenized the markers and role name are combined into 3 tokens. This extra priming is added so that the model knows to produce a reply as coming from the assistant.

Summary of the Logic

• Hidden Formatting: The API does not simply add the raw text as shown—it wraps messages in special tokens that indicate the start/end of messages and encode roles.
• Tokenization is Not Literal: While the string might seem longer when viewed as plain text, the tokenizer (designed to work with subword units) compresses parts of the formatting into fewer tokens.
• Fixed Overhead: For each message provided by the user, there’s a fixed overhead of 4 tokens, and for generating a reply, there’s an overhead of 3 tokens (due to the internal assistant prompt).

These fixed token counts are built into the ChatML encoding format (and thus into the OpenAI API) so that you can predict and manage the total number of tokens in the conversation. This is why you see “magic numbers” like 4 and 3 in the sample code.