How do I configure EKSPodOperator to use in-cluster authentication with a service account on EKS?

Below is some guidance on troubleshooting this issue when using the EKSPodOperator with in‑cluster authentication on EKS.

---

1. Use Only In‑Cluster Authentication

The error

​ "Invalid connection configuration. Options kube_config_path, kube_config, in_cluster are mutually exclusive. You can only use one option at a time."

indicates that more than one authentication configuration is being provided. When a pod is running in the cluster, the expectation is that the Kubernetes client will detect the in‑cluster configuration automatically via the mounted service account token and CA certificate. Make sure your connection does not mix in‑cluster settings with other authentication options. For example, your connection URL should use only a single option:

YAMLCopied
kubernetes_default: kubernetes://?extra__kubernetes__namespace=airflow&extra__kubernetes__in_cluster=True

Double‑check that no additional fields such as kube_config or kube_config_path are inadvertently included in the connection (for example, via environment variables or defaults injected by another mechanism).

---

2. Remove Conflicting or Default Connections

Since you mentioned that you didn’t create the kubernetes_default or aws_default connections yourself, review the following:

• Airflow Secrets Backend:
  If your AWS Secrets Manager backend is configured to load default connection definitions, it might be providing conflicting data. Verify your secrets stored in AWS do not add extra parameters that conflict with the in‑cluster configuration.

• Airflow Helm Chart Defaults:
  The official Helm chart sometimes creates default connection definitions. Check that these default connections are not setting values (like kube_config_path) that may conflict with your in‑cluster configuration.

If possible, remove or override these defaults so that the only active settings for the Kubernetes connection are the correct in‑cluster options.

---

3. Validate Service Account Configuration

Since you are aiming to authenticate using the scheduler’s service account (which is annotated with the proper IAM role and policies), confirm that:

• The scheduler pod is indeed running with the annotated service account.
• The service account is correctly configured to access the Kubernetes API in EKS.
• There is no misconfiguration in the IAM role annotation that might force an alternative authentication method.

---

4. Test with a Minimal Pod Manifest

To isolate the issue, try deploying a simple pod (without using the EKSPodOperator) that relies on the same service account. A manifest similar to the one below can help ensure that the in‑cluster configuration works as intended:

YAMLCopied
apiVersion: v1
kind: Pod
metadata:
  name: test-in-cluster-auth
spec:
  serviceAccountName: your-scheduler-service-account  # Use the service account with the proper IAM role
  containers:
  - name: test-container
    image: yourTestImage
    command: [ "sh", "-c", "echo Hello in cluster && sleep 3600" ]

If that pod starts correctly and can access Kubernetes via in‑cluster config (e.g., by running kubectl get pods internally), then you know the base authentication setup is correct.

---

5. Review EKSPodOperator Code and Parameters

The EKSPodOperator should support in‑cluster authentication if no conflicting parameters are present. Ensure that:

• You are not inadvertently passing any deprecated or conflicting parameters to the operator.
• Any operator defaults are aligned with your intended configuration.

Check the operator’s documentation or source code (of your installed version) to verify that it uses only the in‑cluster configuration when appropriate.

---

Summary

To successfully use in‑cluster authentication with EKSPodOperator on EKS:

1. Ensure your Kubernetes connection string uses only the in‑cluster option (with no kube_config or kube_config_path parameters).
2. Remove or override any default/extra connection definitions that might be injected by the AWS secrets backend or Helm chart.
3. Verify that your scheduler’s service account is properly annotated for IAM roles.
4. Test the in‑cluster configuration with a simple pod to isolate the issue.
5. Check the operator’s parameters to avoid conflicts.

Following these steps should help resolve the authentication conflict and allow your tasks to run using the in‑cluster service account on EKS.