01.10.2025 Problem:
A Prometheus deployment failed a security compliance scan when Prisma Cloud flagged a high-severity vulnerability in the sidecar container image kiwigrid/k8s-sidecar:1.30.9. The issue was linked to the pip package, which was identified as vulnerable to arbitrary code execution. This raised concerns about both compliance and the security of the monitoring stack.
Process:
Step 1: Reviewing the Compliance Scan
Prisma Cloud flagged pip v25.2 inside the sidecar image. The report associated this package with a known arbitrary code execution vulnerability, creating an immediate compliance failure for the deployment. Since Prometheus relies on the sidecar for configuration reloading, addressing this risk was critical to maintain both functionality and security.
Step 2: Mapping the Vulnerability
The vulnerability was matched to advisories affecting pip versions 24.1b1–25.0. These versions were documented as vulnerable to arbitrary code execution when handling package installations. This confirmed that the Prisma finding was legitimate and not a false positive, requiring direct remediation.
Step 3: Verifying the Scope
Further analysis determined that the issue did not impact Prometheus itself but was isolated to the kiwigrid/k8s-sidecar container. This distinction was important, as it allowed the team to address the vulnerability by targeting the sidecar image without needing to alter Prometheus core components.
Step 4: Identifying the Root Cause
The investigation revealed that the sidecar image was built using Python and included pip as part of its build process. Since the package was left inside the runtime image instead of being removed or replaced during hardening, it became a persistent source of risk. Newer image releases addressed this by either updating or stripping pip, confirming that upgrading would mitigate the issue.
Solution:
The recommended solution was to upgrade the sidecar image to kiwigrid/k8s-sidecar:1.31.0 or newer, where the vulnerable version of pip had been removed or patched. For Prometheus environments managed via Helm, the fix involved updating the values file with the patched image tag and applying a rolling upgrade, ensuring zero downtime. As an alternative, teams could adopt hardened images such as Chainguard’s k8s-sidecar, which exclude unnecessary build tools by design, or rebuild the image themselves and explicitly remove pip during the build process. Each approach restores compliance, mitigates the arbitrary code execution risk, and strengthens the overall security posture of the Prometheus deployment.
Conclusion:
By upgrading to a patched or hardened sidecar image, the Prometheus deployment resolved the high-severity vulnerability flagged by Prisma Cloud and restored compliance. The case highlights the importance of monitoring not just core applications but also supporting containers, where overlooked build-time tools like pip can introduce significant risks. Addressing the issue ensured both security and reliability without service disruption.