Optimizing Apache Cassandra Repair: Reducing CPU Utilization from 90% to 30%

Problem:

The client reported high CPU utilization (up to 90%) across all nodes in their 3-node Apache Cassandra 4.1.3 cluster during full or incremental repair operations initiated from any single node. Despite relatively low data volumes (~25 GB per node), the CPU spike raised concerns about system performance, stability, and potential downtime during repairs.

Process:

Step 1: Initial Assessment

1. The expert requested cluster details, including compaction strategy, JVM configuration, heap size, Cassandra version, nodetool settings, and CPU graphs.
2. The client confirmed default compaction strategy (SizeTieredCompactionStrategy), compaction throughput of 16 MB/sec, 1 concurrent compactor, Cassandra version 4.1.3, and JVM using CMS (`-XX:+UseConcMarkSweepGC`).

Step 2: Diagnosis and Recommendations

1. The expert explained that high CPU is expected during repairs due to compaction and network I/O, and offered optimization guidance:
• Compaction Tuning: Confirm workload pattern (write-heavy vs. time-series), but default strategy (STCS) was acceptable for now.
• JVM Tuning: Recommended using G1GC instead of CMS for heaps larger than 12 GB; Cassandra 4.x defaults to G1GC, but client was still using CMS manually.
• Heap Sizing: Advised setting explicit heap values in `cassandra-env.sh` with a maximum of 50% of node RAM, minimum 2 GB.
• Repair Strategy: Explained that running `nodetool repair` on one node also repairs ranges it shares with others, leading to duplicate work. Advised using `nodetool repair -pr` (primary range only) per node to reduce load.

Step 3: Follow-Up & Testing

1. The client ran repair with the `-pr` flag and observed significant improvement: CPU dropped from 90% to 30%, and repair duration was reduced.
2. They requested guidance on selecting heap values based on system RAM and clarification on how to properly switch to G1GC.
3. They also requested a list of best-practice configuration parameters for `cassandra.yaml` to support a growing cluster (~40 GB per node).

Solution:

1. Used `nodetool repair -pr` to prevent redundant range repairs and reduce resource consumption.
2. Recommended setting `MAX_HEAP_SIZE` to no more than 50% of node RAM and `HEAP_NEWSIZE` according to application needs, with reference to Cassandra documentation.
3. Confirmed that G1GC is preferred over CMS for heaps >12 GB and provided guidance for enabling it if needed.
4. Suggested monitoring tools (Prometheus + Grafana + Cassandra Exporter) for improved observability of GC and system metrics.
5. Shared common tuning areas for `cassandra.yaml`: networking (`listen_address`, `broadcast_rpc_address`), cluster metadata (`cluster_name`, `seeds`), and security (`authenticator`, `authorizer`). Advised keeping most other settings at default unless scaling issues emerge.

Conclusion:

Through a combination of targeted configuration changes and operational best practices, the client significantly reduced CPU utilization during repair operations and gained a deeper understanding of Cassandra internals. By switching to `nodetool repair -pr`, optimizing heap size and considering a switch to G1GC, they improved performance, reduced risk of downtime, and are now better equipped to scale their Cassandra environment.