12.07.2024 Problem:
The customer is experiencing the “exec format error” issue when using Prometheus node exporter versions 1.5.0 and 1.6.0 on ARM64 machines, particularly Graviton-type instances in an AWS environment. This error is observed in the node exporter pods running as a DaemonSet in a Kubernetes cluster with nodes having ARM64 architecture.
Process:
The experts requested the information for the investigation.
Step 1:
Experts requested the full name of the container image, which the client using for node_exporter. The team provided the instructions to the customer on how to get it:
‘To get the info please use: $ kubectl get daemonset/daemonset-name -o jsonpath=”{..image}’
Step 2:
After the investigation, the expert’s team suggested to the client to use the image prom/node-exporter-linux-arm64:v1.6.0 instead of prometheus/node-exporter:v1.6.0, because they used ARM64 instance and the docker image version of node exporter should be ARM64 also.
Solution:
To address this issue, experts were recommended to use the “prom/node-exporter-linux-arm64:v1.6.0” image, which was specifically built for ARM64 architecture. This image is expected to resolve the “exec format error” problem and ensure the proper functioning of the node exporter on ARM64-based nodes.
Conclusion:
The “exec format error” issue encountered with Prometheus node exporter versions 1.5.0 and 1.6.0 on ARM64 machines, specifically on AWS Graviton instances, was due to an incompatibility between the Docker image and the ARM64 architecture of the nodes. The problem was identified in a Kubernetes cluster where node exporter pods were deployed as a DaemonSet on ARM64 nodes.
Upon investigation, it was discovered that the customer was using the generic prometheus/node-exporter:v1.6.0 image, which is not designed for ARM64 architecture. The experts recommended switching to the prom/node-exporter-linux-arm64:v1.6.0 image, which is specifically built to support ARM64 architecture.