Configuring n8n HTTP Request Node for SendGrid Unsubscribe Pagination

Below is one approach you can take to let n8n handle SendGrid’s offset-based pagination correctly.

Option 1: Offset Pagination with Query Parameter

1. Set Pagination Mode:
   In the HTTP Request node, first select Add Option > Pagination. Then choose the mode Update a Parameter in Each Request.

2. Configure Query Parameters:

   • Make sure that you have enabled Send Query Parameters in the main node settings.
   • Set the Name of the query parameter to "offset".
   • For Value, toggle on the Expression switch and enter:

   JAVASCRIPTCopied
   {{ $pageCount * 500 }}

   Since $pageCount starts at zero, the first request sends offset=0, the second sends offset=500, and so on.

3. Make Sure to Set the Limit:
   While the pagination settings only update the offset, also include the query parameter limit (set to 500) as a fixed value so each request is correctly limited:

   • Under Send Query Parameters, add another parameter with:

     Name: limit
     Value: 500

4. Notes:

   • If you see the same data repeatedly, double-check that SendGrid isn’t caching responses and that the API really uses the provided offset.
   • Some endpoints might ignore the offset if additional adjustments are required in the request structure. In that case, verify your request by testing the URL manually.

Option 2: URL Pagination Leveraging the Link Header

SendGrid returns a Link header with several URLs (next, prev, etc.). If you prefer to extract and use the next URL directly, do the following:

1. Set Pagination Mode:
   In the node’s Pagination settings, choose Response Contains Next URL.

2. Extracting the Next URL:
   In the Next URL field, use an expression that parses the link header and returns the URL for rel="next". For example, you could use a regular expression to extract the URL:

   JAVASCRIPTCopied
   {{
     $response.headers.link &&
     $response.headers.link.match(/<([^>]+)>;\s*rel="next"/)
       ? $response.headers.link.match(/<([^>]+)>;\s*rel="next"/)[1]
       : null
   }}

   This expression checks if the link header exists and then extracts the URL that has the rel="next" relation. If it finds no next URL, it returns null and pagination stops.

3. Notes:

   • Ensure that your API call contains the correct header in the response for the expression to work.
   • If the header contains multiple entries, using a regex like the one above helps isolate the next page URL.

Summary

• For Offset Pagination:
  Use Update a Parameter in Each Request with the query parameter "offset" set using the expression {{ $pageCount * 500 }} and include the fixed limit=500 parameter.

• For URL Pagination:
  Use Response Contains Next URL and set the Next URL field to an expression that extracts the URL with rel="next" from the Link header.

Either approach will let you paginate through SendGrid’s unsubscribe endpoint. Choose the one that best fits your scenario and testing results.