How can I structure context documents so gpt-3.5-turbo accurately recognizes specific fields?

One approach is to structure your context in a way that clearly delineates each field. Here are a few suggestions:

1. Use JSON Format

If possible, format your context documents as JSON. The structure makes it easy for the model to recognize field names. For example:

JSONCopied
{
  "documents": [
    {
      "title": "Drupal AI SolrAI - CSS",
      "content": "blah, blah…",
      "eventDate": "",
      "documentGroups": "Documentation, Development, D9 Development, AI",
      "documentTaxonomy": "",
      "documentSummary": "blah, blah…",
      "url": "https://master1and1-9.schoolboard.net/node/1445"
    },
    {
      "title": "AI: Drupal SolrAI Module",
      "content": "blah, blah…",
      "eventDate": "",
      "documentGroups": "Documentation, Development, D9 Development, AI",
      "documentTaxonomy": "AI ToDo",
      "documentSummary": "blah, blah…",
      "url": "https://master1and1-9.schoolboard.net/node/1402"
    }
  ]
}

2. Introduce Clear Newlines and Field Separators

If JSON isn’t an option, add newlines and consistent markers so each field is on its own line. For example:

PHPCopied
$documentString = "Document Title: {$documentTitle}\n";
$documentString .= "Content: {$contextDocument}\n";
$documentString .= "Event Date: {$documentDate}\n";
$documentString .= "Document Groups: {$documentGroups}\n";
$documentString .= "Document Taxonomy/Tags: {$documentTaxonomy}\n";
$documentString .= "Source Document Summary: {$documentSummary}\n";
$documentString .= "URL: {$documentURL}\n";
$documentString .= "-----\n";

This formatting helps the model pick out titled fields, as each field consistently starts at a new line.

3. Consider Using Markup or Bullet Points

Another formatting variation is using bullet points or numbered lists. This can also improve field recognition:

PHPCopied
$documentString = "• Document Title: {$documentTitle}\n";
$documentString .= "• Content: {$contextDocument}\n";
$documentString .= "• Event Date: {$documentDate}\n";
$documentString .= "• Document Groups: {$documentGroups}\n";
$documentString .= "• Document Taxonomy/Tags: {$documentTaxonomy}\n";
$documentString .= "• Source Document Summary: {$documentSummary}\n";
$documentString .= "• URL: {$documentURL}\n";
$documentString .= "-----\n";

4. Provide an Instruction in the Prompt

Include a short instruction telling the model how to interpret the format. For example:

The following context documents are formatted so that each field starts on a new line and is labeled. When answering the question, consider the field "Document Title" as the title of each document.

Summary

Using structured data (like JSON) or ensuring that each field begins on its own line with consistent labels should help gpt-3.5-turbo recognize the title field more accurately. While gpt-4 might handle less-structured text better, providing clear and predictable delimiters can improve the performance of earlier models.

Feel free to try these options to see which best meets your needs.