How can I resolve aggregation issues with a custom metric shared across two GCP projects?

When you write an identically named custom metric (with the same labels) from two different projects, Cloud Monitoring sees them as coming from distinct "sources" (i.e. different monitored resource contexts). Even though the metric type and labels match, the project (or more generally, the monitored resource) becomes part of the identity. This is why when you view the metric without filtering, it appears "split" even though filtering by project reveals that each project is writing essentially the same metric.

What’s Happening

• Separate Time Series: Metrics in Cloud Monitoring are uniquely identified not just by their metric type and labels but also by their associated monitored resource (such as the project). Writing the same metric in two projects produces two separate time series, even if the metric names and label keys/values are identical.
• Interval Differences: If the metrics are written at slightly different intervals, this separation is even more pronounced. The slight timing differences reinforce that they’re coming from different sources, so Cloud Monitoring can’t aggregate them as a single series.
• Aggregation Challenge: When you query across projects without properly handling differences in the monitored resource (or other labels), you see what looks like a “broken” metric since it’s actually two separate series.

How to Fix It

If your intention is to have a single unified metric, you have a couple of options:

1. Consolidate Metric Writing:
   Make sure that you write the custom metric data from a single project rather than from multiple projects. This ensures that all data points go to the same monitored resource and time series.

2. Merge or Relabel During Query:
   If consolidating the metrics isn’t possible, consider adjusting your queries (using Cloud Monitoring’s capabilities or exporting/aggregating the data) to merge the data from the two sources. For example, you might use aggregation functions or relabel the time series in your query to combine them. Note that this approach may not be as straightforward because the underlying time series still remain separate.

3. Adjust Resource Labels:
   If the duplication is accidental, verify your configuration so that when you intend a metric to be global (if that’s supported), you’re not inadvertently scoping it to a project. Some custom metrics can be defined at a global level, but this depends on the service and configuration used.

Conclusion

The “broken” appearance of the metric is due to having the same custom metric data written from two different projects at slightly different times, creating two separate time series. Depending on your needs, you can either consolidate the metric writing to a single project or adjust your queries to merge the separate time series.