Optimizing Cassandra Cluster Configuration for Massive Data Ingestion

Problem:

The client requested to review cluster configuration and advise any changes to the configuration parameters to avoid any potential issues proactively. Additionally, the client requested advice on how to identify Cassandra database cluster’s workload.

Process:

1) Data Collection:

• Gathered configuration files
• Collected the last 5,000 lines of the system log

2) Expert Review:

• Conducted a thorough examination of the provided configuration files
• Analyzed system logs to identify any inconsistencies or issues

Solution:

Analysis of the client files revealed the stability of the current configuration, but the following actions are recommended to reduce the probability of possible issues:

• Manually set MAX_HEAP_SIZE in cassandra-env.sh
• If the OS instance has 16+ GB RAM, set MAX_HEAP_SIZE to 8 GB (or half of available RAM). By default, Cassandra takes 1/4 of available RAM
• If the OS instance has 60+ GB RAM, do not set MAX_HEAP_SIZE to more than 30 GB. CMS GC is not designed to work well with Heaps over 30 GB

In order to identify the Cassandra database cluster’s workload, the client should have an appropriate monitoring solution. Cassandra provides its metrics via JMX. To enable JMS, the client should make the following changes in cassandra-env.sh and restart Cassandra:

JMX_PORT="7199"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
    

Alternatively, they can use a tool named Jolokia. It exposes JMX to HTTP JSON. Instructions for monitoring client requests are described in the Cassandra documentation. Search for “Client Request Metrics” on the page. The request types include CASRead, CASWrite, Read, and Write.

Conclusion:

The case involves a production data migration to Cassandra and seeks expert guidance to review the cluster configuration. The configuration files appear to be mostly default and robust for handling the migration, but a proactive recommendation is made to adjust the MAX_HEAP_SIZE in cassandra-env.sh for potential issues during massive data ingestion. Specific guidelines for setting MAX_HEAP_SIZE based on available RAM are provided. The case also emphasizes the importance of implementing monitoring solutions, suggesting the use of JMX to access Cassandra metrics and explaining how to enable it. Alternatively, the case introduces the option of using the Jolokia tool for the same purpose. Lastly, it directs attention to monitoring client requests, providing instructions on tracking various request types for a successful data migration.