How do I permanently resolve the disk-pressure taint on my Kubernetes node running containerd?

The node.kubernetes.io/disk-pressure taint is applied automatically by the kubelet when it detects that one or more disk metrics (like imagefs.available, imagefs.inodesFree, nodefs.available, or nodefs.inodesFree on Linux) are below the thresholds set for the node. Simply removing images or trying to manually remove the taint without addressing the underlying disk pressure won’t work because the kubelet continues to monitor the actual disk usage and will reapply the taint as long as disk pressure persists.

Here are some steps to resolve the issue:

1. Free Up Disk Space on the Node

• Identify Heavy Consumers: Use tools like du to check which directories are consuming most space (e.g., container logs, stray temporary files, or large images).

  BASHCopied
  sudo du -sh /* 2>/dev/null

• Clean Up Logs and Temporary Files: For instance, rotate or clean up large logs found in /var/log or any application-specific logs.

• Prune Unused Data: While you already ran crictl rmi --prune and crictl rm -a, verify if there are other caches or files related to containerd that could be cleared.

2. Check Kubelet Disk Threshold Settings

Review the kubelet configuration to see if the disk pressure thresholds are too aggressive for your workload. The kubelet uses flags like:

• --eviction-hard
• --eviction-soft
• --eviction-soft-grace-period

Adjusting these might help in cases where occasional spikes in usage are causing the taint. However, the safest approach is to ensure sufficient disk space.

3. Address Underlying Disk Usage Issues

• Expand Disk Capacity: If disk utilization is consistently high, consider increasing the disk capacity or adding extra storage.

• Optimize Workloads: Ensure that pods, especially those with high ephemeral storage usage, are scheduled appropriately. If a pod requires or tolerates disk pressure, consider adding a pod-level tolerance for disk pressure, but only after ensuring that the node’s disk pressure situation is under control.

4. Monitor the Node After Cleanup

After cleaning up disk space, monitor the node’s disk metrics:

BASHCopied
kubectl describe node foo-node1

The taint should eventually disappear once the underlying metrics are above the threshold. If the node still shows the taint, re-check disk usage and threshold configurations.

---

Key Takeaway:
The disk-pressure taint remains until the kubelet sees that the disk usage has returned above the configured low thresholds. Tactical manual taint removal (e.g., kubectl taint node foo-node1 node.kubernetes.io/disk-pressure-) will only work temporarily if the disk pressure condition still exists. Focus on resolving underlying disk consumption issues to permanently remove the taint.