Does GPT-4-128K Accurately Retrieve Middle-Embedded Information from Long Contexts?

Below is a concise summary of the experiments and results described:

---

Background Context

• Prior Research Findings:

  • Some studies (e.g., “Lost in the Middle: How Language Models Use Long Contexts”) have reported that language models tend to perform best when relevant information is located at the beginning or end of the context.
  • Performance reportedly drops when key information is buried in the middle of very long contexts (e.g., between 50%–70% depth in 60K+ token documents).

• Needle in the Haystack Test:

  • This experiment involves embedding a specific sentence (the “needle”) within a large text (the “haystack”), then querying the model to access that fact.
  • Previous tests indicated a drop in performance when the needle was placed in the middle of long contexts.

---

The Hypothesis

• Core Assumption:
  Increasing the signal strength—by duplicating the target statement (needle)—might help the model retrieve it even if it is placed in the middle of a long context.

• Additional Considerations:

  • The method of using the GPT-4 API directly (instead of an intermediary library such as LangChain) may also influence performance.

---

Experiment Details

1. Experiment 1: Two Needle Strategy

   • Process:
     • Background tokens were obtained (using concatenated Paul Graham essays, up to 120K tokens).
     • The target statement (“The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.”) was placed twice within various positions (depths ranging from 20% to 80%).
     • The query was: “What is the most fun thing to do in San Francisco?”
     • Answers were generated via GPT-4-Turbo-128K and then evaluated using GPT-4.
   • Result:
     • Achieved 100% accuracy in retrieving the target statement.

2. Experiment 2: Single Needle Strategy

   • Process:
     • The procedure was similar, except that only one instance of the target statement was embedded.
   • Result:
     • The retrieval rate was still 100% accuracy across the tested ranges (context sizes 60K–120K tokens and depth from 20% to 80%).

---

Key Takeaways

• Reinforcing the Signal:

  • Doubling the needle clearly reinforces the signal, ensuring retrieval with 100% accuracy.
  • Even a single injection of the target statement (when using the GPT-4 API directly) achieved perfect retrieval, contradicting earlier studies that reported challenges with “middle of context” retrieval under similar conditions.

• Methodology Impact:

  • The difference between these results and earlier experiments (which used libraries like LangChain) may be due to:
    • The strength of the embedded signal.
    • Variations in how the text is processed when using the GPT-4 API directly compared with intermediary libraries.

• Implications for Future Evaluations:

  • The way input is structured (e.g., signal reinforcement, direct API calls) can significantly affect performance in retrieval tasks.
  • There might be opportunities to improve long context retrieval performance by strategically placing or reinforcing important information within the context.

---

Conclusion

The experiments challenge the notion that LLMs uniformly struggle to retrieve information from the middle of long contexts. Rather, with careful design—such as reinforcing key information—the GPT-4 family of models can achieve perfect retrieval even when the key information is deeply embedded. This insight is important for both evaluating long-context models and designing systems where information retrieval from extensive documents is needed.

---