Clarification on Assistants API Pricing and Token Counting

Below is an overview of how pricing and token usage works with the Assistants API:

When Are You Charged?

• Per Run: Charges occur when a Run is invoked. A Run is an invocation of the Assistant on a Thread. In other words, when the Assistant processes a conversation turn (making calls to models and tools) that constitutes a Run, the tokens involved in that Run (both input and output) are billed.
• Messages Added vs. Run Invocation: Adding a message to a Thread does not itself trigger billing. It’s when that message is used in a Run (i.e. when the Assistant reads from the Thread and responds) that the associated tokens are billed.

How Tokens Are Calculated

• Input and Output Tokens: Like other OpenAI API calls, the total token count is the sum of tokens in the input sent to the model and the tokens in its output.
• Entire Thread in a Run: For each Run the Assistant processes, tokens are counted for the messages that are actually incorporated into the call. If the assistant builds its context from the thread’s history, then effectively the tokens from that history included in the Run are charged.

Handling Long Threads and Truncation

• Automatic Truncation: When a thread becomes too long for the model’s context window, the system will truncate older messages. This truncation happens behind the scenes.
• Token Billing Implication: Since only the tokens sent to the model count, tokens from truncated parts aren’t included in the current API call—and therefore are not billed again in that particular run.

Knowledge Retrieval and Token Pricing

• Inclusion in the Run: If your Assistant performs knowledge retrieval and then uses that retrieved information as part of the conversation (for example, by injecting it as context into a Run), those tokens are counted as part of the model call.
• No Separate Token Calculation: There isn’t a separate mechanism for token calculation for knowledge retrieval; it’s integrated into the overall count for the API call.

Estimating Token Usage Before a Run

• Use the Tokenizer: OpenAI provides a Tokenizer tool which you can use to check how a string is split into tokens.
• tiktoken Library: For programmatic estimation, the tiktoken library is available, making it easier to simulate token counts based on your prepared text.
• Pre-run Estimation: Before each Run, you can manually (or programmatically) combine the messages that will be sent and estimate the tokens, ensuring you remain within the model’s limits and have an idea of the cost.

Summary

• You are charged based on each Run, not just when the message is appended to the thread.
• The tokens are computed for each Run from the input and output text that is actually processed.
• Automatic truncation of long threads ensures that only the tokens included in the model call are billed.
• Knowledge retrieval tokens are included in the same way once they form part of the context.
• Tools like the Tokenizer and tiktoken library help you estimate tokens before making a Run.

This setup leverages the backend to manage heavy lifting with token management, though it’s important to keep an eye on the messages included in each Run to manage costs effectively.