Why Is the Executor Pod Template Ignored in Spark 3.3.1 on Kubernetes?

Below is some guidance for troubleshooting executor pod template issues when running Spark on Kubernetes:

1. Verify Template Accessibility and Correctness

Ensure that the executor pod template file is accessible from the specified S3 location and is correctly formatted as a Kubernetes pod spec. For example, confirm that the file includes the proper volume mounts and container port definitions for JMX. You might run a quick check by manually downloading the file from S3 and validating its YAML syntax.

YAMLCopied
apiVersion: v1
kind: Pod
metadata:
  name: spark-executor-template
spec:
  containers:
    - name: spark-executor
      ports:
        - containerPort: 9010   # Example JMX port
      volumeMounts:
        - name: pod-template-volume
          mountPath: /opt/spark/pod-template
  volumes:
    - name: pod-template-volume
      configMap:
        name: your-configmap

2. Spark Version and Feature Support

Double-check that the Spark version you are using has full support for both driver and executor pod templates. Some versions or configurations may support one and not the other. The driver pod template working while the executor template is not applied could be an indication of a version-specific or feature-specific limitation.

3. How Executor Pods are Launched

In some Spark-on-Kubernetes patterns, the driver pod acts as a controller and is responsible for launching executor pods. Because of this, the executor pod templates must be applied during that launching process. Confirm that the Spark code or submission logic is correctly picking up the value provided in the property:

spark.kubernetes.executor.podTemplateFile=s3a://mybucket/code/exec-pod-template.yaml

Review logs from both the driver and the resulting executor pod creation process. Look for any errors or warnings related to reading or merging the pod template.

4. Consistency with Driver Template

Since you mentioned that the driver pod template works (appearing at /opt/spark/pod-template), compare the driver and executor templates. Look for discrepancies that might cause the executor template to be ignored. For instance, the executor template may be missing required sections or may have specification differences that prevent proper merging.

5. Spark JIRA and Community Guidance

You've referenced SPARK-51020 via JIRA, which suggests that you aren’t the first to encounter this behavior. Keep an eye on that ticket for workarounds or patches. Additionally, it may be helpful to review the Spark and Kubernetes communities’ discussions for any Spark-specific guidance regarding executor pod templating.

By carefully checking the template file, Spark configuration, and executor launch process, you should be able to pinpoint why the executor template is not being applied. Remember that the integration between Spark patterns and Kubernetes pod management can sometimes require extra configuration steps compared to the driver pod.

If issues persist after these checks, providing log excerpts and a detailed configuration comparison between driver and executor environments might help others give more targeted advice.