How do I fix the 'Forbidden' error in n8n's HTTP Request node when accessing The Hacker News Vulnerability page?

Below are some steps you can try to resolve the "Forbidden" error:

1. Use a Modern Browser User-Agent

Even if you’ve added a User-Agent header, double-check you’re using one common for browsers. For example:

JSONCopied
{
  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
}

2. Include Other Browser Headers

Sometimes websites don’t just check the User-Agent; they also look at other standard headers. Try adding headers such as:

JSONCopied
{
  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
  "Accept-Language": "en-US,en;q=0.5",
  "Connection": "keep-alive"
}

In n8n’s HTTP Request node, you would add these headers alongside your GET request configuration.

3. Enable Following Redirects

Some sites may push you through redirects to a location that then requires more headers. Check the HTTP Request node settings to ensure that “Follow Redirects” is enabled.

4. Check for IP Blocking or Bot Protection

The Hacker News site might be employing measures to block automated requests even if they don’t require login credentials. You might try:

• Using a proxy or changing the request’s origin IP.
• Experimenting with delays or limiting the frequency of requests.

5. Verify Request from a Browser

Open your browser’s developer tools (Network tab) and inspect the exact headers your browser is sending when accessing the page. Replicate those headers in your HTTP Request node.

Example n8n HTTP Request Node Configuration

Here’s a rough example of how you might set the headers in n8n:

JSONCopied
{
  "requestMethod": "GET",
  "url": "https://thehackernews.com/search/label/Vulnerability",
  "headers": {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Connection": "keep-alive"
  },
  "followRedirect": true
}

Recap

• Verify you’re using a valid, modern User-Agent.
• Add additional headers to better mimic a genuine browser request.
• Ensure that redirects are followed.
• Consider if the site might be blocking based on IP or bot behavior.

By making your HTTP request look more like a standard browser request, you increase the chances of bypassing the bot protection measures in place on The Hacker News site.