How Do GPT Models Generate String-Level Embeddings from Token-Level Embeddings?

When we say that GPT models are based on token embeddings, that refers only to the very first step in the network: converting each token (subword) into a vector. However, the magic happens in the subsequent layers. Here’s how the process works:

1. Contextualization through Self-Attention:
   Once tokens are embedded, many transformer layers (using self-attention) process these embeddings. This means each token’s representation becomes context-dependent, as it starts incorporating information from its surroundings.

2. Aggregation into a String-Level Representation:
   For tasks like embedding a whole sentence or document (which is common in RAG architectures), the individual token representations are combined into a single vector. This can be done using methods like:

   • Pooling: Averaging (or taking other forms of pooling) the token embeddings to generate a single vector.
   • Using a Special Token: Some models use a designated token (like the CLS token) whose representation is designed to capture the overall meaning of the entire sequence.

3. Semantic Similarity:
   The resulting vector for the whole string represents the aggregated meaning of the input text. This makes it possible to compute similarity metrics (such as cosine similarity) between different strings, even though the underlying model started with token-level embeddings.

In summary, while the GPT architecture begins with token embeddings, the network transforms these through multiple layers, culminating in a final vector that captures the overall semantics of the string. The OpenAI embeddings API leverages this by returning these aggregated, semantic-rich vectors, enabling effective similarity comparisons for entire texts.