Cassandra JMX Exporter Misconfiguration Leads to Prometheus Latency Issues

Problem:

The client wanted to integrate Apache Cassandra metrics into Prometheus for monitoring and alerting. While using JMX Exporter with Node Exporter, they noticed that too many unwanted metrics were being fetched. This resulted in performance issues such as delayed response times from the curl command and unnecessary load on Prometheus. Additionally, key metrics like cassandra_up and cassandra_status were not available, and they wanted guidance on either customizing Prometheus or using scripts to expose these metrics reliably.

Process:

Step 1: Initial Analysis

The expert reviewed the existing config.yaml file used by the JMX Exporter and identified that the configuration lacked filtering mechanisms such as whitelistObjectNames or rules tailored to the client’s required metrics. The file collected an excessive number of metrics, overwhelming Prometheus and slowing metric retrieval to over 13 seconds. Some critical Cassandra metrics were missing from the output entirely.

Step 2: Metric Requirement Assessment

The client shared a list of high-priority metrics, including:

• cassandra_clientrequest_latency
• cassandra_clientrequest_timeouts
• cassandra_cache_hitrate
• cassandra_table_readlatency
• cassandra_table_writelatency
• cassandra_threadpools_pendingtasks
• cassandra_storage_load

The expert proposed limiting the scope of collected metrics by specifying only the required MBeans in whitelistObjectNames, and building minimal regex patterns in the rules section to parse them efficiently.

Step 3: Configuration Optimization

A revised configuration was shared with reduced metrics and targeted rule patterns. The following optimizations were applied:

• Use of whitelistObjectNames to explicitly allow only relevant Cassandra MBeans.
• Defined simple rules to match metrics like Latency, Timeouts, HitRate, and Load.
• Excluded JVM metrics that were not required using excludeJvmMetrics: true.
• Upgraded to JMX Exporter version 1.3.0 to ensure compatibility with modern features like includeObjectNames.

Step 4: Live Testing and Iteration

Multiple meetings were conducted with the client to test different versions of the configuration. Performance improvements were observed — response time dropped to around 9 seconds. However, some metrics remained missing due to typos in regex patterns or misidentified MBeans. These were corrected live during collaborative sessions, and final configs were exported for production and PET (pre-production) environments.

Step 5: Custom Metrics and Monitoring Advice

Since Cassandra does not natively expose cassandra_up and cassandra_status via JMX, the expert confirmed that a custom exporter or external script would be required. The expert explained that implementing this falls outside standard support and may require a separate custom development contract.

Solution:

The expert provided a working JMX Exporter configuration that limited Prometheus ingestion to essential Cassandra metrics. The revised YAML file used whitelistObjectNames and simplified rules to ensure efficient metric parsing. This solution reduced Prometheus metric load and improved response times significantly.

The expert also clarified paths forward for exposing additional health-check-style metrics like cassandra_up, recommending custom script integration if needed. With these changes, the client could continue Prometheus integration in production with minimal overhead and better clarity over monitored metrics.

Conclusion:

The integration of Cassandra metrics with Prometheus initially overwhelmed the client’s monitoring system due to an overly broad JMX Exporter configuration. By narrowing down the exported metrics and optimizing the Prometheus scraping configuration, the expert delivered a streamlined and efficient monitoring solution. While some metrics required further customization, the core integration was stabilized and ready for production rollout. The expert’s involvement helped the client achieve their goals with measurable performance improvements and minimal disruption.