How can I fix intermittent RedisCluster connection issues with ioredis on GCP?

IAM auth token is a short live token, it is valid for one hour only (in GCP context).

Meaning, if one of your connections disconnected for any reason after an hour, you need to regenerate a new access token and use it as the password.

Even if you didn't have a disconnection, the authenticated connection is valid for 12 hours only, and should be re-authenticate.

In addition, GCP Redis cluster closes an idle connection after 10 minutes of inactivity.

So few things happen here:

1. Each time you are idle for more than 10 min, your pods/local connection gets disconnected.
2. If the disconnection happens after more than one hour, you get wrong pass error since the token is not valid anymore.
3. If you don't have any disconnection, but you didn't refresh the connection using a new token, your connection will get kicked out after 12 hours, and you will get again pass err.
4. Even though it is not a little, probably because of limited compute and networking in Cloud Functions, refresh slots take more time, and they are failing to do so on time.

Solutions:

For the idle issue, just send a ping once in a few minutes.

ioredis doesn't support OOB credentials providers, so you will need to set the new tokens to the client-connection object manually, the best solution for all the issues above is to manually schedule a replace each 50 min and re-auth each 10 hours (less than required for safety):

JAVASCRIPTCopied
function renewToken() {  
    // Logic to generate or retrieve the new token  
    return 'yourNewToken'; // Replace with your actual token logic  
}  

async function updatePassword() {  
    const newToken = renewToken();  
    try {  
        // Update the Redis password  
        clusterClient.options.redisOptions.password = newToken;  
        console.log('Password updated successfully to:', newToken);  
    } catch (error) {  
        console.error('Error updating password:', error);  
    }  
}  

async function authenticate() {  
    const newToken = renewToken();  
    try {  
        // Authenticate with the new token  
        await clusterClient.auth(newToken);  
        console.log('Authenticated successfully with new token:', newToken);  
    } catch (error) {  
        console.error('Error during authentication:', error);  
    }  
}  

function schedulePasswordUpdates() {  
    // Initial password update immediately  
    updatePassword();  

    // Update password every 50 minutes  
    setInterval(() => {  
        updatePassword();  
    }, 3000000); // 50 minutes (3000000 milliseconds)  

    // Every 10 hours (600 minutes): update password and authenticate  
    setInterval(async () => {  
        await updatePassword();  
        await authenticate(); // Run authentication after updating the password  
    }, 60000000); // 10 hours (60000000 milliseconds)  
}  

// Start the periodic password update and authentication  
schedulePasswordUpdates();  

See more on automating renew token in GCP docs: https://cloud.google.com/memorystore/docs/cluster/manage-iam-auth#automate_access_token_retrieval.

Other options:

1. You can choose to just replace the token each hour and let it deal with the disconnection each 12 hours by the client retrying automatically using the new token you updated.
2. You can choose not to renew at all and have an error handling, in which each time you get disconnected, you just kill the client, get a new token and recreate the client.

What to consider when choosing from the above:

1. If you get disconnected, reconnect is more costly than simply sending auth, so if it's happening during heavy traffic time, it is not the best idea.
2. If you choose to kill the client and recreate, you lose all the inflight commands that were on their way.

So in general, I recommend the first option.

For the [ioredis] Unhandled error event: ClusterAllFailedError: Failed to refresh slots cache. error, just increase the slotsRefreshTimeout so Cloud Functions has enough time to complete.

Disclosure:

I'm from AWS Elasticache, and not from GCP, or using memory Store.

My knowledge about GCP memory Store and IAM comes from working together with GCP engineers on valkey-glide, and working currently on designing OOB IAM integration for the valkey-glide which will do all the above without the user need to set it all by itself both for GCP and AWS.
And because of the similarities of Elasticache IAM usage and memory store IAM usage.

I might miss something unique to GCP, but I don't think so, my work currently in the design including integration with both, and a nice amount of research on GCP IAM auth.

See GCP pointing to glide as the future client of valkey/redis-oss.