How can I resolve the 403 Forbidden error when using Metrics Server for HPA on EKS Fargate?

Below is some guidance to help troubleshoot your 403 Forbidden error when the Metrics Server tries to scrape metrics from your Fargate pods.

Understand the Error

A 403 error indicates that the request from the Metrics Server to the kubelet (or its equivalent endpoint on Fargate) is being actively rejected. In a traditional self-managed or managed node environment, the kubelet’s authentication and TLS settings (or any RBAC configuration) might need adjustment. In the case of EKS Fargate, it’s important to know that the infrastructure is different compared to EC2-backed worker nodes. Fargate does not publish kubelet endpoints in exactly the same way, so the Metrics Server’s default scraping method might be blocked.

Things to Check

1. Kubelet Endpoint Availability
   On Fargate, the mechanism to expose pod-level metrics may differ. The Metrics Server usually scrapes the kubelet’s resource metrics endpoint on port 10250. With Fargate, confirm whether that endpoint is available or if AWS requires an alternative mechanism (for example, using CloudWatch Container Insights or a custom metrics adapter).

2. Parameters Passed to Metrics Server
   You’re already passing:

   • --kubelet-insecure-tls
   • --kubelet-preferred-address-types=InternalIP

   These flags generally help bypass certificate issues when scraping kubelet endpoints on regular nodes. However, on Fargate the endpoint itself may not be reachable or might reject the request regardless. In addition, note that you’re setting --secure-port=10250, but the logs indicate the server is operating on port 4443. Verify that you’re not mixing arguments intended for the Metrics Server’s own serving endpoint with those used to contact the underlying node (or Fargate) components.

3. Network Policies and RBAC
   Even if the Fargate environment uses a different mechanism for exposing metrics, make sure that no network policies or RBAC settings are inadvertently blocking the Metrics Server from making requests. Although Fargate networking is largely managed by AWS, double-check that your cluster’s policies don’t add extra restrictions.

Recommended Next Steps

• Review Fargate Metrics Capabilities
  Confirm with the current EKS Fargate documentation if pod resource metrics scraped from kubelet endpoints are supported. In many cases, Fargate may require using an AWS-specific method (like CloudWatch Container Insights) or a custom metrics adapter for HPA to work properly.

• Adjust Metrics Server Configuration
  Depending on your investigation, you might need to adjust the Metrics Server’s parameters. If Fargate pods do not expose the kubelet endpoint on 10250 in the same way as traditional nodes, you might have to remove or modify the insecure flags and ensure you’re not overriding safe defaults that are compatible with Fargate.

• Explore Alternative Metrics Adapters
  If it turns out that Fargate does not support the endpoint that Metrics Server targets, consider an adapter that can pull metrics from AWS CloudWatch and expose them via the Kubernetes custom metrics API. This allows HPA to scale based on CPU (or custom) metrics without relying on the default kubelet scrape.

Summary

The 403 Forbidden error suggests that while your Metrics Server is configured to bypass some common TLS issues (with --kubelet-insecure-tls and address type preference), it’s still unable to authenticate against an endpoint that on Fargate might not be available or compatible with the conventional kubelet scrape. Verify the supported metrics model for your EKS Fargate environment and adjust your setup accordingly. This might involve using a different metrics adapter or reconfiguring the Metrics Server to use endpoints that are actually available on Fargate.