How do I resolve token limit errors when my prompt and max_tokens exceed the model's context length?

I am using GPT-3.5 Turbo to break long chapters from my books into smaller segments by processing each CSV row, sending the text combined with a prompt to the API, and then writing the segmented output to a new CSV file.

The problem is that although my input, prompt, and expected output tokens are well within limits (generally 2-3k tokens total), I still encounter an error about the token limit being exceeded. For example, with gpt-3.5-turbo the error states: "This model’s maximum context length is 4097 tokens. However, you requested 5136 tokens (1136 in the messages, 4000 in the completion)..." and for gpt-3.5-turbo-16k it mentions requesting 17521 tokens, which is above its limit.

I have verified that my individual chapter lengths should not surpass 4000 tokens, but it appears that the API is summing the tokens in the prompt (approximately 1136 tokens) and the max_tokens parameter (set to 4000 for gpt-3.5-turbo and 16385 for gpt-3.5-turbo-16k), which together exceed the allowed tokens for these models. I suspect I need to adjust the max_tokens value so that prompt tokens + max_tokens is within the model's context length.

Below is the core code that performs these operations. Any insights on adjusting the token values or other issues in my Python code are appreciated.

PYTHONCopied
import os
import pandas as pd
import openai
from flask import Flask, render_template, redirect, url_for
from dotenv import load_dotenv
from itertools import cycle

app = Flask(__name__)

# Load API key
load_dotenv('path_to_setup.env')
openai.api_key = os.getenv("GPT_API")

@app.route('/')
def process_text(text):
    engine = "gpt-3.5-turbo-16k"
    message = [
        {"role": "user", "content": "PROMPT OF APPROX 750 TOKENS"},
        {"role": "user", "content": text}
    ]
    response = openai.ChatCompletion.create(model=engine, messages=message, temperature=0.5, max_tokens=16385)
    return response['choices'][0]['message']['content']
    
@app.route('/process')
def process_csv():
    # Open or create the output CSV file
    output_file_path = 'path_to_output_file.csv'

    # Read source CSV
    for chunk in pd.read_csv('path_to_input.csv', chunksize=1):
        # Process each row
        for idx, row in chunk.iterrows():
            text = row['Text']
            processed_text = process_text(text)
            parts = processed_text.split('(SEGMENT_END)')  # Splits ouput text at a marker in the text.
            parts = [part.strip() for part in parts if part.strip()]  # Remove extra line breaks and white spaces
            letters = cycle('abcdefghijklmnopqrstuvwxyz')
            for i, part in enumerate(parts):
                # Write data to output
                data = pd.DataFrame({
                    'Level': [row['Level']],
                    'Book': [row['Book']],
                    'Chapter': [f"{row['Chapter']}{next(letters)}"],
                    'Len': [len(part)],
                    'Text': [part]
                })
                # Append this row's processed data to the output CSV
                data.to_csv(output_file_path, mode='a', index=False, header=False, encoding='utf_8_sig')

if __name__ == '__main__':
    app.run(debug=True)