Secure Your Monitoring with the Latest kube-prometheus-stack Upgrade - Proactive Insights and Support For Open-Source Applications

Problem:

The client is using kube-prometheus-stack version 44.2.1 as their monitoring solution, and Twistlock scans have identified vulnerabilities in various packages including github.com/docker/distribution (CVE-2023-2253), golang.org/x/net (CVE-2022-41723), github.com/emicklei/go-restful/v3, and Go language with CVEs such as CVE-2021-29923, CVE-2021-38297, CVE-2021-39293, CVE-2021-41771, CVE-2021-41772, CVE-2021-44716, CVE-2022-23772, CVE-2022-23773, CVE-2022-23806, CVE-2022-24675, CVE-2022-24921, CVE-2022-27664, CVE-2022-28131, CVE-2022-28327, CVE-2022-2879, CVE-2022-2880, CVE-2022-30580, CVE-2022-30630, CVE-2022-30631, CVE-2022-30632, CVE-2022-30633, CVE-2022-30635, CVE-2022-32189, CVE-2022-32190, CVE-2022-41715, CVE-2022-41716, CVE-2022-41723, CVE-2022-41724, CVE-2022-41725, CVE-2023-24534, CVE-2023-24536, CVE-2023-24537, and CVE-2023-24538.

Solution:

To address the identified CVE vulnerabilities in the kube-prometheus-stack, the client updated their Helm chart to versions 46.6.0 and 46.8.0. These versions offer fixes for a comprehensive list of CVEs, including CVE-2021-29923, CVE-2022-23772, and CVE-2023-24538.

The client used the following commands to upgrade:

helm install [RELEASE_NAME] --version 46.6.0 prometheus-community/kube-prometheus-stack
helm install [RELEASE_NAME] --version 46.8.0 prometheus-community/kube-prometheus-stack

During the scanning of Helm chart version 46.8.0, the following images were reviewed for vulnerabilities:

docker.io/bats/bats:v1.4.1
docker.io/grafana/grafana:9.5.3
quay.io/kiwigrid/k8s-sidecar:1.24.3
quay.io/prometheus-operator/prometheus-operator:v0.65.2
quay.io/prometheus/alertmanager:v0.25.0
quay.io/prometheus/node-exporter:v1.5.0
quay.io/prometheus/prometheus:v2.44.0
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2

The client was advised to thoroughly test the new Helm chart version in development, testing, or staging environments before deploying it to production. For CVE-2022-41723, which remains unresolved in the Logstash application (docker.elastic.co/logstash/logstash 8.8.1), the client needs to wait for a patch release from the developer.

Conclusion:

The customer successfully upgraded the kube-prometheus-stack from version 44.2.1 to version 46.8.0 to address multiple vulnerabilities identified through Twistlock scans. This update resolved a wide range of CVEs, significantly enhancing the security and stability of the monitoring solution. The upgrade was implemented using the Helm command, and thorough testing was conducted in a development environment before deploying to production. However, CVE-2022-41723 remained unresolved in the latest Logstash application, awaiting a patch from the developer. Overall, these updates have greatly improved the security posture of the customer’s monitoring setup.