MySQL on GKE Autopilot with GCSFuse: How Can I Resolve '/var/lib/mysql' Permission Denied Errors?

Title:
MySQL on GKE Autopilot with GCSFuse: Permission denied on /var/lib/mysql/

Body:

I am deploying MySQL 8.0 on Google Kubernetes Engine (GKE) Autopilot using Google Cloud Storage (GCS) with GCSFuse as a PersistentVolume. MySQL’s data files are stored in a GCS bucket and mounted to /var/lib/mysql.

However, MySQL fails to start due to permission issues, producing the following error:

BASHCopied
Defaulted container "mysql" out of: mysql, gke-gcsfuse-sidecar (init), fix-permissions (init)
2025-02-04 16:58:35+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.41-1.el9 started.
2025-02-04 16:58:38+00:00 [Note] [Entrypoint]: Initializing database files
mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (OS errno 13 - Permission denied)
2025-02-04T16:58:38.828764Z 0 [ERROR] [MY-010460] [Server] --initialize specified but the data directory exists and is not writable. Aborting.
2025-02-04T16:58:38.828780Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable.
2025-02-04T16:58:38.829179Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.41)  MySQL Community Server - GPL

---

Setup Overview

The deployment setup is as follows:

1. PersistentVolume (PV) and PersistentVolumeClaim (PVC) using gcsfuse-csi:

   • Bucket: www_1
   • Mount Path: /var/lib/mysql
   • Access Mode: ReadWriteMany (RWX)

2. MySQL Deployment Configuration:

   • An initContainer (fix-permissions) that runs chown -R 999:999 /var/lib/mysql to set the correct ownership
   • A sidecar container (gke-gcsfuse-sidecar) to mount the GCS bucket
   • MySQL 8.0 running with a volume mount at /var/lib/mysql

3. Service Account Setup:

   • The GKE service account is linked to a custom service account
   • The service account has all required GCS permissions (roles/storage.admin, roles/storage.objectAdmin)
   • The necessary directories (e.g., databases/api) already exist in the GCS bucket

---

What I Have Tried

• Verifying PVC status:

BASHCopied
kubectl get pvc shared-storage-pvc

Output indicates the PVC is Bound with RWX access.

• Checking PersistentVolume details:

BASHCopied
kubectl describe pv shared-storage-pv

Output confirms the PV is properly attached to the PVC.

• Attempting to inspect MySQL directory permissions (pod fails to start):

BASHCopied
kubectl exec -it deployment/api-mysql-deployment -- ls -lah /var/lib/mysql

• Verifying if GCSFuse is mounted read-only (pod fails to start):

BASHCopied
kubectl exec -it deployment/api-mysql-deployment -- mount | grep /var/lib/mysql

• Reviewing pod logs:

BASHCopied
kubectl logs deployment/api-mysql-deployment

Logs consistently report permission errors at /var/lib/mysql.

I am looking for guidance on correcting these permission issues so that MySQL can initialize properly on GKE Autopilot using GCSFuse for its storage.