Understanding Label Usage in Prometheus Alerts - Proactive Insights and Support For Open-Source Applications

Problem:

The client has encountered an issue where a label, specifically “app_kubernetes_io_part_of”, is not being evaluated in the alert description or labels despite being present in the metric. They seek clarification on whether this behavior aligns with the expected functionality of Prometheus alerts.

Process:

The client provided a Prometheus rule with an alert definition that utilized the app_kubernetes_io_part_of label.

Despite the label being present in the metric data, it did not appear in the alert description or labels.

After modifying the alert expression to include the label in the sum by clause, the label’s value started appearing in the alert description.

The client sought clarification on whether this behavior was expected in Prometheus alerts and whether labels not defined in the alert expression could be used in alert descriptions or labels.

Solution:

The expert confirmed that the observed behavior aligns with Prometheus’s design for label inclusion in alerts. Labels used in alert annotations or labels must be included in the aggregation operator clause of the alert expression. Therefore, only labels included in the sum by or similar clauses can be utilized in alert descriptions or labels.

Conclusion:

This clarification helps ensure the correct usage of labels in Prometheus alerts, providing clarity for future alert rule configurations.