Why is the max_tokens parameter for gpt-3.5-turbo-1106 capped at 4096 output tokens?

Below is an explanation based on the current OpenAI API reference:

Background on max_tokens vs. max_completion_tokens

• Deprecated Behavior:
  Previously, the parameter max_tokens was used with text completions. For the older API endpoints (or older models), the prompt token count plus the specified max_tokens were required to stay within the model’s context length. This is why you might have been able to pass a higher max_tokens value historically.

• Current Behavior for Chat Completions:
  For models such as gpt-3.5-turbo-1106 and other chat models, the parameter now clearly states that max_tokens (or the newer max_completion_tokens) refers only to the number of tokens to generate (i.e., the output tokens). The token count of the prompt is handled separately when enforcing the overall context length limit.

What Changed?

• New Definition for max_tokens:
  The documentation now explicitly defines max_tokens as "The maximum number of tokens to generate in the chat completion."
  This means the prompt tokens are no longer included in this count.

• Impact on Your Usage:
  If you’re trying to set a value like 10,000 for max_tokens and encountering an error, it’s because for chat completions that value is being interpreted as a cap on the output tokens. With models like gpt-3.5-turbo-1106, the maximum limit is 4096 output tokens. The overall conversation (prompt + generated tokens) still has a higher cap (for example, 16,385 tokens), but max_tokens (or max_completion_tokens) is solely for the generated part.

Summary Answer

In short, yes—when using the chat completion APIs, max_tokens now only specifies the maximum number of tokens to generate (output tokens). The prompt tokens are not included in this count. The overall context window still enforces that the sum of input (prompt) tokens and output tokens does not exceed the model’s limit.

If you were previously seeing that token counts included the prompt, it might have been due to using an older endpoint or model behavior. The updated documentation confirms that for chat completions, only the output tokens are controlled by this parameter.