Why does GPT-4 with a 128K input context only generate up to 4096 output tokens?

Below is a brief explanation with supporting details from our documentation:

Why Only 4096 Output Tokens?

The behavior you’re observing is expected for the model you’re using. While your model supports a 128K token input context, the output (or completion) is capped at a lower limit—in this case, 4096 tokens. This isn’t an error in your experiment or code but rather a design choice in the model’s configuration.

How Token Limits Work

The model has two separate constraints:

• Input Context Limit: Determines how many tokens you can send in your prompt (or conversation history). For your model, this is 128K tokens.
• Output (Completion) Limit: Sets the maximum number of tokens the model can generate in a single completion. For “gpt-4-1106-preview”, this is currently limited to 4096 tokens.

Any attempt to bypass this by setting a higher max_tokens value will result in an error indicating that the model supports at most 4096 tokens for completions.

What Can You Do?

If your experiment requires translations that would generate more than 4096 tokens, consider one of these approaches:

• Chunking the Task: Break your input into smaller segments that can be translated separately, then reassemble the output.
• Chaining Requests: Use multiple API calls where the output from one is fed as input to the next, effectively "continuing" the translation across calls.

Summary

The limitation you’re encountering is an intentional model constraint rather than an issue with your implementation. The model’s design allows for very large input contexts while capping the number of tokens it generates to ensure efficient and reliable operation.

Let me know if you have any more questions on fine-tuning or using the OpenAI API!