How Is Token Usage Calculated When Using the ChatGPT API for Assistants?

Below is an explanation of what goes into the token calculation for ChatGPT API calls:

What Counts as Tokens

1. Input tokens:
   Every piece of text you send to the API—whether it's the main prompt, helper or system instructions (like your 5-paragraph intro), or any additional context—is converted into tokens. If you’re sending the same 5-paragraph intro with every API call, then yes, it is re-counted each time.

2. Output tokens:
   The text generated by the model is also tokenized, and each token in the response counts toward your bill. This means that even if you send a fixed amount of input text, a lengthy output will increase your token count and thus the cost.

How Token Usage Affects Cost

• Per-call token count:
  Your billing is based on the combined total of input and output tokens. For example, if your API call sends 50 tokens and receives 150 tokens, you are billed for 200 tokens.

• Different pricing per model:
  GPT-4 typically has a higher cost per 1,000 tokens compared to GPT-3.5. If you were previously using GPT-3.5 with lower rates (leading to charges such as $1.50 or $2 per call) and then switched to GPT-4 at about $0.03 per 1,000 tokens, the increase in cost—even for similar token counts—would be substantial.

Your Scenario

• Assistant’s 5-paragraph intro text:
  If this introductory text is sent as part of every prompt you send to the API, it counts as input tokens on every call.

• Content text per request:
  The 1 paragraph (up to 8000 characters) of your actual content on every request will also be tokenized. Both the number of tokens from this content and any tokens generated in the response add to your total.

• Example Calculation:
  If the intro text uses, say, 100 tokens and your content text uses 200 tokens, each call has 300 input tokens. If the model then responds with 300 tokens, the call uses 600 tokens. At GPT-4 pricing, every additional token makes a difference.

Summary

• Every text element of your API call, whether part of the system prompt, user prompt, or assistant message, is converted into tokens.
• Input tokens (which include your consistent 5-paragraph intro) are included in the billing calculation for every request.
• Output tokens (the generated text) are also part of the calculation.
• Your overall cost is determined by the sum of input and output tokens multiplied by the price per 1,000 tokens for the model in use.

For further clarity, you can use tools like the Tokenizer tool or the tiktoken library to count tokens in your text strings before making API calls. This can help you optimize your design and better forecast costs.

Let me know if you have any further questions about using the OpenAI API!