How can I debug gRPC 'Call Cancelled' errors potentially caused by a GKE internal load balancer?

Below is some guidance based on the Kubernetes documentation regarding Services of type LoadBalancer on GKE:

Logging and Modifying the LB

Kubernetes creates a cloud-managed load balancer when you specify a Service of type LoadBalancer (with the internal annotation in this case). The configuration parameters you can set via the manifest are mostly limited to what is supported in the Kubernetes API (such as annotations and the deprecated loadBalancerIP). There isn’t a documented way to enable detailed load balancer logging or modify the internal health check configurations directly through the Kubernetes manifest.

For enhanced logging or to inspect load balancer behavior, you’ll need to rely on the cloud provider’s tools—in this case, GCP Cloud Logging. Keep in mind that modifications to LB logging or health check parameters are generally done through the GCP console or via provider-specific configuration mechanisms rather than Kubernetes manifests.

Regarding the gRPC "Call Cancelled" Error

Since the load balancer created for a Service of type LoadBalancer in GKE (with the internal annotation) acts in a pass-through manner, it does not normally manipulate individual gRPC connections. However, the underlying load balancer’s health checks, timeouts, or even provider-specific behaviors (such as connection draining) might induce cancellations if endpoints become temporarily unhealthy. The Kubernetes docs note that load balancer health checks are critical to managing traffic, but the exact implementation of these checks is defined by the cloud provider.

Next Steps

1. Review GCP Logs:
   Check Cloud Logging for your load balancer to see if there are any time-correlated events (such as health check failures or reconfigurations) that match the twice-a-day cancellations.

2. Examine Health Check Settings:
   Although not directly configurable via the manifest, review your GCP backend configuration associated with the internal load balancer. Look for any configured timeouts or health check intervals that might be affecting idle connections.

3. Investigate Client/Server Behavior:
   While it seems likely that the load balancer might be involved based on your investigation, do ensure that your gRPC client and server aren’t inadvertently triggering cancellation due to timeout or connection management settings.

4. Provider-Specific Annotations (if needed):
   If further customization is needed, see if GCP supports additional annotations for load balancer configuration. The Kubernetes documentation suggests that in cases requiring custom LB behavior, cloud-provider–specific annotations are the way to go.

Summary

In short, there isn’t a documented way to enable or modify LB logging for the internal load balancer directly within the Kubernetes manifest. You should rely on GCP’s Cloud Logging and review the health check configuration on the GCP side. This approach will help you confirm whether the load balancer is the source of the cancellations, or if it’s necessary to further investigate the gRPC client/server implementations.