How can I handle tool calls with streaming responses in GPT-4-1106-preview?

Below is an overview of the new approach with GPT-4-1106-preview when handling tool calls with streaming. Instead of manually checking for a "function_call" key in each streamed chunk, the API now aggregates tool calls and signals them via a finish_reason (often "tool_calls"). That means you no longer need to piece together arguments from chunks in the same way.

What's Changed

• No More Partial "function_call" in Deltas:
  Rather than receiving partial function call information in each chunk, the system now groups any tool calls in the response. The API eventually returns a finish_reason (e.g., "tool_calls") indicating that a set of tool calls should be performed.

• Event-Driven Handling:
  For streaming responses, the documentation now suggests using an event handler (such as in the Python or Node SDKs) to capture when the assistant indicates a required action. For instance, you can subclass an AssistantEventHandler to listen for events like "thread.run.requires_action", and then process tool calls in bulk.

• Processing Multiple Tool Calls:
  With the new parallel tool-calling support, the API may return multiple tool calls in one response. Each call is now organized in an array (typically under a field like tool_calls), and you need to iterate through this array to process each call.

Example with EventHandler

Below is a simplified example (adapted from the docs) for handling streaming responses with multiple tool calls:

PYTHONCopied
from typing_extensions import override
from openai import AssistantEventHandler

class EventHandler(AssistantEventHandler):
    @override
    def on_event(self, event):
        if event.event == 'thread.run.requires_action':
            run_id = event.data.id
            self.handle_requires_action(event.data, run_id)

    def handle_requires_action(self, data, run_id):
        tool_outputs = []

        # Iterate through the list of tool_calls provided in the event
        for tool in data.required_action.submit_tool_outputs.tool_calls:
            if tool.function.name == "get_current_temperature":
                tool_outputs.append({"tool_call_id": tool.id, "output": "57"})
            elif tool.function.name == "get_rain_probability":
                tool_outputs.append({"tool_call_id": tool.id, "output": "0.06"})

        # Submit the aggregated tool outputs back to the API
        self.submit_tool_outputs(tool_outputs, run_id)

    def submit_tool_outputs(self, tool_outputs, run_id):
        # Use the submit_tool_outputs_stream helper from the client
        with client.beta.threads.runs.submit_tool_outputs_stream(
            thread_id=self.current_run.thread_id,
            run_id=self.current_run.id,
            tool_outputs=tool_outputs,
            event_handler=EventHandler(),
        ) as stream:
            for text in stream.text_deltas:
                print(text, end="", flush=True)
            print()

# Now create a streaming session that uses the event handler.
with client.beta.threads.runs.stream(
    thread_id=thread.id,
    assistant_id=assistant.id,
    event_handler=EventHandler()
) as stream:
    stream.until_done()

Key Points

• Finish Reason as Indicator:
  When you see a finish_reason like "tool_calls", it means that the assistant’s response includes one or more tool calls that you need to process. This replaces the old method of manually assembling function call arguments during the stream.

• Parallel Function Calling:
  The new API supports sending multiple tool calls in parallel. For each tool call in the array, you would handle them (for example, by executing the respective function and later sending back an updated message with the results).

• Streaming Remains Active:
  While streaming, you continue to receive output like text deltas. However, the tool call portion is now sent as a complete set (typically signaled as part of a terminal event), so there's no need to continuously append partial arguments from each chunk.

Final Thoughts

If you previously relied on checking each delta for a "function_call" key in your stream, you'll now need to transition to using the event handling approach, checking for action-required events, and handling the complete tool call set as shown above.

This should help you adapt your code to work effectively with the new GPT-4-1106-preview model and its streaming behavior.