How can I use WebSockets with the OpenAI API for real-time client-server communication?

Below is some guidance on using WebSockets with OpenAI's API for real-time communication.

OpenAI Realtime API

OpenAI provides a dedicated Realtime API designed for stateful, event-based real-time communication over a WebSocket. It isn't the same as the standard HTTP endpoint and comes with its own documentation.

Key Connection Details:

• URL:
  wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01

• Headers:
  You need to include the following headers when establishing the connection:

  • Authorization: Bearer YOUR_API_KEY
  • OpenAI-Beta: realtime=v1

Example Using Node.js with the ws Library

Below is an example code snippet demonstrating how to connect:

JAVASCRIPTCopied
import WebSocket from "ws";

const url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01";
const ws = new WebSocket(url, {
  headers: {
    "Authorization": "Bearer " + process.env.OPENAI_API_KEY,
    "OpenAI-Beta": "realtime=v1",
  },
});

ws.on("open", function open() {
  console.log("Connected to server.");
  ws.send(JSON.stringify({
    type: "response.create",
    response: {
      modalities: ["text"],
      instructions: "Please assist the user.",
    }
  }));
});

ws.on("message", function incoming(message) {
  console.log(JSON.parse(message.toString()));
});

Where to Find the Documentation

The details for connecting using WebSockets and the event-based nature of the communication are outlined in the OpenAI Realtime API documentation. Here’s the link for further reading:
OpenAI Realtime API Guide

This documentation covers the necessary parameters, including the WebSocket URL, headers, query parameters, and the different client events that the server accepts.

Summary

For your use case involving real-time client-server communication, you can use the OpenAI Realtime API over WebSocket by following the configuration shown above. Make sure to use the proper headers and endpoint as described in the documentation. If you need further details about specific events or advanced usage, refer to the documentation linked above.

Let me know if you have any other questions related to this topic!