Is the GPT-4 preview model enforcing a lower effective token limit than expected?

Below is a brief explanation with reference to how the OpenAI API and its parameters work:

---

What’s Likely Happening

The new preview model appears to enforce limits on the number of prompt tokens more strictly. The error message indicating inability to provide direct quotes is consistent with the model ending a run early when it determines that the input exceeds the allowed token usage for a single run—exactly the behavior described for the API’s token management.

In the API reference documentation, there’s a parameter called max_prompt_tokens. This parameter lets you set the maximum number of prompt tokens that may be used over the course of a run. If a run’s cumulative prompt tokens exceed this threshold, the system terminates the run with an “incomplete” status. Although you mentioned the overall prompt had 61,169 tokens (and the output was only a few hundred tokens), if the internal limit is lower than what you expect (or if default limits are applied in the preview model), similar behavior can occur.

---

Why Did the Behavior Change?

• Model Differences: The gpt-4-0125-preview model might have different internal token thresholds compared to previous models (like turbo). Even if your prompt worked fine in a longer instruction version, the new model might apply stricter checks that trigger when it sees a very large number of tokens.

• Internal Enforcement: The preview model likely enforces the upper limit of allowed prompt tokens during the entire run. When your prompt approaches or exceeds the internal limit, the model refuses to provide direct quotes or content as a safeguard (as described in the documentation).

• Prompt Structure Variation: Sometimes, even small changes in the instructions (for example, more succinct phrasing) can alter the model’s interpretation of token boundaries or how it performs internal token counting. That might explain why the longer prompt ran without issue but the shorter version triggered these messages.

---

Key Takeaways

• Check max_prompt_tokens: The API reference clearly states that if the specified (or default) token limit is exceeded, the run returns an incomplete status along with messages like the ones you observed.

• Model-Specific Behavior: The preview version may have different default settings or stricter limits than earlier models. This is not just a hallucination – it’s an enforcement of internal token limits to ensure stable and predictable performance.

• Managing Input Size: To avoid this behavior, either consider splitting your input into smaller chunks or explicitly set/adjust the max_prompt_tokens parameter (if your API version and model allow it). This will ensure that the run stays within the allowed token usage.

---

In summary, this isn’t a bizarre hallucination. The model’s response is likely due to internal processing limits related to the maximum number of prompt tokens allowed in a run. Adjusting your workflows to respect these limits should help you avoid receiving these “incomplete” responses.