Why Does the gcp_billing_export Table Not Show All GKE Namespace Workloads?

Below is an explanation of why you might see differences in namespace data between the two tables:

Summary

The primary (gcp_billing_export) table is designed to capture cost data at a higher abstraction level, while the gke_cluster_usage_consumption table is tailored to capture detailed metering data, including all workload namespaces with additional usage fields.

Detailed Explanation

• gcp_billing_export Table

  • This table collects cost data that comes with its own set of metadata.
  • It may not always include labels for every workload namespace because cost attribution might aggregate or filter out some label details depending on usage events and billing record processing.
  • This table is primarily focused on providing a summary of costs (like the queries that sum the cost and group by cluster names).

• gke_cluster_usage_consumption Table

  • This table is created when you enable GKE Metering and is specifically designed to provide deeper insight into your GKE usage.
  • It includes detailed columns (beyond what’s in the main billing export table) that allow for a workload-level view, including all namespaces along with additional usage metrics.
  • It serves as the primary source for granular insights (like detailed breakdown for workloads namespace-wise).

Key Points from the Documentation

• The GCP documentation states that namespace data should be present in both tables. However, due to the differences in how data is collected and processed:
  • Not every workload namespace label might be attached in the gcp_billing_export table because its main focus is on cost aggregation.
  • The gke_cluster_usage_consumption table captures a broader set of data (workload-specific details and extra columns) exactly for scenarios like detailed usage and consumption analysis.

What You Can Do

If your goal is to analyze workloads namespace-wise with full detail, use the gke_cluster_usage_consumption table, as it guarantees the inclusion of all namespaces and offers additional columns that provide more context for your usage data.

Example Query for Namespace Extraction (From gcp_billing_export Table):

SQLCopied
SELECT DISTINCT namespace
FROM `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`,
UNNEST(labels) AS labels
WHERE labels.key = "k8s-namespace";

Note: For in-depth namespace analysis, consider querying the gke_cluster_usage_consumption table where you’ll have all the required workload details.

Conclusion

The observed discrepancy is due to the intended design of each table. The gcp_billing_export table provides an aggregated view of cost-related data, whereas the gke_cluster_usage_consumption table is optimized for detailed workload insights, including a complete breakdown by namespaces.