Is it Worth Switching from Client to Cluster Mode in a Standalone Apache Spark on Kubernetes?

Below is some guidance from a Kubernetes production perspective, though note that the Kubernetes documentation doesn’t prescribe specific recommendations for Spark’s client versus cluster deployment modes.

Considerations for Client vs Cluster Mode

• Fault Tolerance & Resource Management:
  In cluster mode, the driver runs inside the cluster. This can improve resilience and simplify resource accounting because Kubernetes manages both the driver and executors. However, if you have already engineered your client mode setup to avoid transferring large data loads to the driver, that benefit might not be as compelling.

• Operational Impact:
  Switching deployment modes may require refactoring parts of your deployment. If your current setup (with Spark running as a StatefulSet) meets your needs and you’re planning to eventually move to an operator or a native Kubernetes integration, an intermediate switch might add extra work without significant operational gains.

• Future Transition Plans:
  Since you have a roadmap toward using a dedicated operator or native Kubernetes support, it might be more efficient to postpone changes. This would allow you to leverage the improved integration and management options provided by those tools without performing an intermediate refactoring.

Summary

While cluster mode can offer benefits such as centralized management and improved fault tolerance, if your current client mode setup effectively avoids driver-related bottlenecks, there's no inherent Kubernetes-driven advantage in making an immediate switch. Given your goal to minimize extra refactoring, it might be best to maintain your current implementation until you are ready to use an operator or a more integrated Kubernetes approach.

This approach aligns with production considerations in Kubernetes deployments by balancing operational overhead with the benefits of managed services.