How can I prevent PostgreSQL pod restarts during GKE autoscaling?

Below is some guidance based on Kubernetes’ best practices for stateful workloads and Pod Disruption Budgets:

Understand the Limitation with a Single-Replica StatefulSet

When you only have one pod for a stateful workload such as PostgreSQL, no Kubernetes-native mechanism can “live migrate” that pod between nodes. Even with a Pod Disruption Budget, if there’s only one replica nothing can really block an eviction required for a node drain. In other words, the PDB settings are only advisory when it comes to voluntary disruptions—even more so for a single replica.

Why Is the Pod Restarting?

GKE autoscaling that adds a node may trigger node draining procedures or rescheduling events. Since your PostgreSQL StatefulSet is the only instance available, it has to be terminated and recreated despite having a PDB configured. Kubernetes simply does not support moving the running process “in flight” to another node without a restart.

Best Practices and Recommendations

1. Highly Available Configuration
   Best practice for production databases is to run them in a highly available configuration. By deploying multiple replicas (preferably in a clustered mode with a quorum), you can:

   • Use a PDB to ensure that a quorum of pods is maintained during disruptions.
   • Avoid the single point of failure where autoscaling events cause a full service interruption.

2. PDB with maxUnavailable=0
   If you must run a single-instance database for some reason, you have a couple of options:

   • Do not use a PDB so that you explicitly accept downtime.
   • Or, set a PDB with maxUnavailable=0 but note that this won’t prevent an eviction—rather, it adds an extra level of caution. However, this setting requires external coordination (i.e., someone must intervene or manually remove the PDB to allow for a controlled disruption).

3. Node Affinity or Anti-Affinity Adjustments
   While setting node affinity can pin your PostgreSQL pod to a specific node to prevent it from moving around during autoscaling, this limits the flexibility of the autoscaler. You need to balance the requirement for stability against the benefits of fully automated autoscaling.

4. Consider Storage and Restart Impacts
   Since your storage is remaining intact, ensure you are using volumes (e.g., PDs in GKE) that reattach quickly. Even then, the time it takes to restart your PostgreSQL container and rejoin it to your application may be significant. In many cases, this is why most recommendations point toward a multi-replica or HA setup.

Conclusion

Kubernetes does not provide a mechanism to “evade” a required pod restart during node scaling events, especially for a single-instance StatefulSet. The recommended approach is to redesign your deployment for high availability by having multiple replicas (and configuring a proper PDB to maintain quorum), thereby eliminating the single-point failure and mitigating the disruptive effects of node autoscaling. Otherwise, you must accept that with a single replica there will be unavoidable downtime whenever the pod is rescheduled.

Implementing these best practices will help ensure that your stateful workloads remain available even when infrastructure changes occur.