Does the Assistants API count tokens from the entire conversation history with each new message?

Below is an explanation of the key differences regarding token consumption between the two APIs:

Chat Completions API

• History as Input: When using the Chat Completions API, you are responsible for including the entire conversation history (user messages, assistant replies, system instructions, etc.) in every API call.
• Token Consumption: This means that every past message you include counts as part of the input tokens for that call. As a result, if your conversation grows long, you’ll see an increase in prompt tokens, which directly affects billing and efficiency.

Assistants API

• Internal Memory: With the Assistants API, the chat history is automatically managed by the system. You don’t have to resend the earlier messages; the assistant “remembers” the conversation context.
• Token Consumption: Only the tokens from the new input (and the resulting output) are tallied against your usage quota. The internal context is maintained as cached memory, and while it may be tracked using internal metrics (e.g., “cached_tokens”), these cached tokens are not added to the token count in the same way as direct input tokens.
  • In other words, past messages are stored and used to generate context but do not incur additional cost as separate input tokens with every new API call.

Summary

The Assistants API handles conversation history internally without requiring you to manually resend earlier messages for context. Consequently, token usage and costs are determined by the new interaction's prompt and the generated response, rather than the entire historical conversation. This design can help manage efficiency and cost compared to the Chat Completions API, where the full chat trail is directly included in each call.