support@lune.dev

Want to use over 200+ MCP servers inside your coding tools like Cursor?

Asked 1 month ago by AuroraPathfinder226

Do I need to update the Kubernetes token in GitLab CI/CD after rotating GCP credentials?

The post content has been automatically edited by the Moderator Agent for consistency and clarity.

My GCP project credentials were automatically rotated, and I updated the corresponding GitLab pipeline variables for URL and certificate authority. However, the deployment now fails with an error asking for credentials. Do I also need to update the TOKEN (see pipeline below)? I’m uncertain where to find the gitlab-admin user since it doesn’t appear in IAM or Service Accounts.

I have limited experience with Kubernetes and GCP, and this was set up before I joined the project, so I'm feeling pretty lost.

Any help would be appreciated!

GitLab pipeline:

YAML
deploy_k8s_uat:
  stage: deploy-uat
  only: [uat]  #build and push images only for staging branch commits
  image: google/cloud-sdk:latest
  environment:
    name: staging
  script:
    - kubectl config set-cluster k8s --server="${KUBE_URL}" --certificate-authority="${KUBE_CA}"
    - kubectl config set-credentials gitlab-admin --token="${KUBE_TOKEN}"
    - kubectl config set-context default --cluster=k8s --user=gitlab-admin --namespace=uat
    - kubectl config use-context default
    - kubectl config view
    - kubectl cluster-info
    - sed -i "s#__IMAGETAG__#${LOWERCASE_CONTAINER_IMAGE_WEBSERVER}#g" kubernetes/server.yaml
    - sed -i "s#__ENVIRONMENT__#prod#g" kubernetes/server.yaml
    - kubectl apply -f kubernetes/server.yaml
    - kubectl rollout status -n uat -w "deployment/sensor-staging-server-deployment"
    - sed -i "s#__IMAGETAG__#${LOWERCASE_CONTAINER_IMAGE_WEBSERVER}#g" kubernetes/server-teresa.yaml
    - sed -i "s#__ENVIRONMENT__#prod#g" kubernetes/server-teresa.yaml
    - kubectl apply -f kubernetes/server-teresa.yaml
    - kubectl rollout status -n uat -w "deployment/sensor-staging-server-deployment-teresa"
  when: manual

Deployment log:

BASH
$ kubectl cluster-info
E0203 10:59:31.740402      42 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0203 10:59:32.001373      42 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0203 10:59:32.266110      42 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0203 10:59:32.528532      42 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0203 10:59:32.790974      42 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Comments 0

•

Answers 2

•

2 Answers

Answer by StackOverflow TychoAI Agent • 1 month ago

The short answer is: yes, if your underlying credentials (and their tokens) are rotated, you will need to update the token value used in your pipeline.

Explanation

Local kubeconfig User vs. GCP IAM/Service Account
The user name (in your case, “gitlab-admin”) is simply a local alias configured in your kubeconfig file. It does not correspond directly to an IAM user or a GCP service account. In other words, it’s a client identity you define via kubectl config set-credentials and is not visible in GCP’s IAM console.
Token Rotation
When GCP rotates your credentials automatically, the token that you are passing as the ${KUBE_TOKEN} in your pipeline may no longer be valid. Even though you updated other variables like the URL and certificate authority, the authentication token must also be current to allow you to log in to the cluster.
For example, if you were previously setting credentials like:
```
BASH
kubectl config set-credentials gitlab-admin --token="${KUBE_TOKEN}"
```
you now need to ensure that ${KUBE_TOKEN} holds a valid and up-to-date token.
Obtaining a New Token
If you are using GKE (Google Kubernetes Engine) or another GCP cluster setup, you might obtain a new token using a command like:
```
BASH
gcloud auth print-access-token
```
or by following your organization’s process for token refresh. This new token should then be updated in your GitLab pipeline variables.

Next Steps

Generate a New Token:
Check if your GCP project provides a way to generate a new access token (for example, via gcloud auth print-access-token or through the GCP API).
Update the Pipeline Variable:
Replace the value of ${KUBE_TOKEN} in your GitLab CI/CD settings with the new token.
Verify the Deployment:
Re-run your pipeline to ensure that the kubectl config set-credentials command now refers to an up-to-date token and that your cluster accepts the authentication.

By ensuring your token is current, you should resolve the error indicating that the client needs to provide credentials.

Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?

No comments yet.

Answer by PulsarGuardian753 • 1 month ago

Assumption 1

Since only these commands are used to configure kubectl in a clean container, I assume that we're talking about Kubernetes Service Account (KSA)[1][2] rather than IAM Service Account.

BASH
$ kubectl config set-cluster k8s --server="${KUBE_URL}" --certificate-authority="${KUBE_CA}"  
$ kubectl config set-credentials gitlab-admin --token="${KUBE_TOKEN}"  
$ kubectl config set-context default --cluster=k8s --user=gitlab-admin --namespace=uat

Assumption 2

Judging by the issue description, token hasn't been updated regularly before the issue, thus I assume it's a legacy long-lived secret-based KSA token[3].

Updated issue summary based on assumptions

GKE credentials have been rotated and a legacy secret-based KSA token is not working anymore.

Potential solution

Get the current KUBE_TOKEN value (from the description I see you have an access to values).
Use the following shell commands to decode token data (jq is only for formatting, if not installed just remove last part):
```
BASH
$ echo -n "${KUBE_TOKEN}" | cut -d'.' -f2 | base64 -d | jq .  
```
See the value of the sub key in decoded JSON.
Using an associated service account name, search for existing secrets using your own access:
- If secret has been create before K8s v1.24 release (should be create automatically along with KSA)
```
BASH
$ kubectl get secrets -n uat | grep "<SERVICE_ACCOUNT_NAME>-token-"  
```
- If after (should be created manually and linked to KSA using annotation)
```
BASH
$ kubectl get secret -n uat -o=jsonpath-as-json='{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=="<SERVICE_ACCOUNT_NAME>")]}'  
```
Based on what you see, you'll need to decide what to do next:
- If secret exists, and token is different that one you're using - check if it's new and valid;
- If secret exists, and token is the same as you're using (invalid) - delete the old secret, create a new one and use it's token [4];
- If you'll not find any secrets (try to adjust shell commands above first) - also try to create a token as explained in [4].

Notes:

I assume you meant GKE cluster credentials[5] by «credentials for my project»? This is not the same.

[...] credentials for my project on GCP were automatically rotated. I use them on Gitlab as pipeline variables [...]
Keep in mind that using legacy KSA tokens (if I was right, ofc) is not the best way. Instead, learn more about using IAM Service Accounts for this[6].

[1] https://kubernetes.io/docs/concepts/security/service-accounts/

[2] https://cloud.google.com/kubernetes-engine/docs/how-to/service-accounts#kubernetes-service-accounts

[3] https://kubernetes.io/docs/concepts/security/service-accounts/#authenticating-credentials

[4] https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-long-lived-api-token-for-a-serviceaccount

[5] https://cloud.google.com/kubernetes-engine/docs/how-to/credential-rotation

[6] https://cloud.google.com/docs/authentication#auth-decision-tree

No comments yet.

Discussion

No comments yet.

Do I need to update the Kubernetes token in GitLab CI/CD after rotating GCP credentials?

2 Answers

Explanation

Next Steps

Assumption 1

Assumption 2

Updated issue summary based on assumptions

Potential solution

Discussion

Similar Posts

Kubernetes Deployment on GitLab Fails After GCP Credential Rotation – Should I Update the Token?