Why Does GPT-4-1106-Preview Only Generate 4096 Output Tokens Despite a High Maximum Length Setting?

Below is an explanation on what's happening with the token limits for the gpt-4-1106-preview model:

Output Tokens Limit vs. Context Window

• Context Window:
  The figure you saw in the playground (119999) likely refers to the overall context window available for the conversation. Some GPT‑4 variants have large context windows (up to 128k tokens for certain GPT‑4 Turbo models), which means you can send in a lot of input tokens.

• Max Output Tokens:
  However, there is a separate limit on the number of tokens the model can generate in its reply. For gpt-4-1106-preview, this output cap is currently 4096 tokens. In your tests, when you set a large overall length or input a message with 14K tokens, the model’s reply gets truncated because it cannot exceed that 4096 output token limit.

Clarifying the Discrepancy

• Playground vs. API Behavior:
  The playground might present an option for a maximum overall token count that includes the large context window, but in the API (and underlying model logic), the maximum tokens for the completion part remain 4096. This means that even if you see a higher “maximum_length” setting, the generated output will be limited by the model's 4096 token constraint.

• Future Adjustments:
  The documentation and recent blog posts indicate that the stable production-ready models might have some refinements in the future. Until then, even preview versions like gpt-4-1106-preview maintain these limits.

Summary

For the gpt-4-1106-preview model:

• Total context window: May be large (as seen with the high number in the playground)
• Maximum output tokens: Limited to 4096 tokens

The behavior you observed is expected for this version, and if you test via the CLI you should find the same limit on the number of tokens for generated output.

Feel free to ask if you have further questions about managing tokens or other API behavior!