How can I prune chat messages to avoid the 4096 token limit in the OpenAI API?

Below is a summary of some ideas you might consider testing in your lab. Note that many developers work around the hard 4096-token limit by both removing less-relevant messages as you’re doing (e.g. strip out assistant responses) and by applying strategies that alter the conversation history to preserve context while reducing its weight. Here are a few suggestions:

---

1. Summarization of Older Content

Rather than entirely stripping out older messages, consider summarizing them. The process would look something like:

• When the message array exceeds a predetermined token threshold, run those older messages through a summarization routine.
• Replace a batch of messages with a single “summary” message that captures key context.
• For example, system and assistant roles might be summarized together to retain key guidance while cutting excess detail.

Example pseudocode:

RUBYCopied
def summarize_messages(messages)
  # Assume 'llm_summarize' is a function that takes a chunk of text and returns a summary.
  conversation_text = messages.map { |msg| "#{msg[:role]}: #{msg[:content]}" }.join("\n")
  summary = llm_summarize(conversation_text)
  [{ role: "system", content: "Conversation summary: #{summary}" }]
end

# When token count is too high:
if token_count(messages) > MAX_TOKENS_ALLOWED
  # Summarize the oldest half of the conversation.
  summary_messages = summarize_messages(messages[0..(messages.size/2)])
  messages = summary_messages + messages[(messages.size/2 + 1)..-1]
end

This approach reduces tokens while keeping important context intact.

---

2. Selective Role-Based Pruning

You’re already looking at role-specific stripping (assistant, system, or user). Depending on your application’s nuances you might:

• Strip Assistant Only: Useful if the answers aren’t needed for context.
• Strip System and Assistant: If instruction context is clear from earlier interactions, you might remove both.
• Hybrid Approaches: For instance, always keep the latest few messages from each role and prune earlier ones.

Think about interdependencies—often, assistant messages reinforce prior user instructions. Methods like “strip_assistant_and_system” may work well in settings where user turns alone are enough to maintain context.

---

3. Sliding Window with Dynamic Capping

Implement a sliding window mechanism for conversation history:

• Always include the most recent N messages (or N tokens).
• For older parts, either drop or summarize them.
• You can test a variant that “slices” the conversation array at a specific index rather than filtering by role.

Example approach:

RUBYCopied
def prune_sliding_window(messages, token_limit)
  # Keep messages from the end until the total token count is below the limit.
  pruned_messages = []
  total_tokens = 0
  
  messages.reverse_each do |msg|
    msg_tokens = count_tokens(msg[:content])
    break if (total_tokens + msg_tokens) > token_limit
    pruned_messages.unshift(msg)
    total_tokens += msg_tokens
  end
  
  pruned_messages
end

This method ensures you always provide the most recent context given the token budget.

---

4. Token Reduction via Preprocessing

Before sending any conversation history to the API, consider preprocessing steps such as:

• Text normalization: Removing extraneous punctuation or formatting that may add tokens unnecessarily.
• HTML/Markdown stripping: Like cleaning data before passing it to the model.

The OpenAI guides recommend such filtering for large contexts (as seen with techniques like filtering context input).

---

5. Configurable Token Limits per Run

Remember that if you’re working with the OpenAI Assistants API, you have options like specifying max_prompt_tokens and max_completion_tokens at runtime. While this won’t solve the overall context size issue, it can help ensure that each individual run processes an optimal token count. For example:

RUBYCopied
run_options = {
  max_prompt_tokens: 500,
  max_completion_tokens: 1000,
  messages: pruned_messages
}

This kind of configuration works well in tandem with a pruning method and can provide graceful degradation (e.g., if a completion reaches the token limit, you can detect and handle it).

---

Final Thoughts

Experiment with combinations of these approaches. In many cases, a hybrid solution that both summarizes aborted context and applies a sliding window based on token count will yield consistency. The results will help keep each API call within the 4096-token window while preserving the quality of responses.

Give these methods a try in your lab, and best of luck in optimizing your conversation pruning strategy for the new chat API!