Mitigating Disk Overflow and Repair Issues in Apache Cassandra DR Setup

Problem:

The client encountered a critical issue in their Apache Cassandra setup: the Disaster Recovery (DR) cluster was consuming significantly more filesystem (FS) space than the primary cluster, despite having the same data replication strategy. As a result, one of the DR nodes hit 100% disk usage, causing the replica to go down. The client sought support to understand the root cause and resolve the issue without adding disk space.

Process:

Step 1: Initial Assessment

The expert examined the setup using `nodetool status` and confirmed that while the primary cluster nodes had disk usage around 330–345 GiB, the DR nodes ranged up to 470 GiB. One DR node had gone down due to full disk, causing inconsistencies across the cluster. The primary suspect was old or residual data bloating the DR nodes.

Step 2: Disk Cleanup

The expert advised the client to perform the following commands on all Cassandra nodes (primary and DR):

• nodetool cleanup — to remove keys that no longer belong to the node after topology changes
• nodetool clearsnapshot — to delete old snapshots and free up space

Additionally, the expert warned that disk usage exceeding 80% on Cassandra nodes is unsafe and may lead to compaction failures or data inconsistencies. The client acknowledged that both clusters had disks nearing or exceeding this threshold.

Step 3: Node Recovery and Repair Strategy

After running cleanup commands, disk usage reduced significantly on the previously full node, allowing it to rejoin the cluster. The client then initiated a full repair across the DR cluster.

However, repair failed with the following error:

Repair job has failed with the error message: Repair command #1 failed with error Endpoint not alive: /100.122.56.26:7050

This occurred because one of the DR nodes was still down, and Cassandra could not complete the repair while a required endpoint was unreachable.

Step 4: Node Rejoin and Final Repair

The expert advised that the failed node should be removed and re-added to the cluster to avoid inconsistencies. Once all nodes were in the UN (Up/Normal) state, a full repair should be initiated again.

The client expressed concern about potential data loss from node removal. The expert clarified that as long as a full repair is performed after the rejoin, no data loss should occur due to Cassandra’s replication and hinted that this process is safe if done carefully.

Solution:

The expert guided the client through a multi-step process that included cleanup, snapshot removal, rejoining failed nodes, and finally a successful full repair. Key actions taken:

• Executed nodetool cleanup and nodetool clearsnapshot on both clusters
• Waited for failed DR node to rejoin successfully
• Performed full repair only after all nodes were marked UN
• Advised long-term solution: adding new nodes or increasing per-node disk size

The DR cluster’s disk usage normalized after cleanup, and the final repair succeeded without further incident.

Conclusion:

This incident highlighted how Cassandra clusters can diverge in disk usage due to residual data and failed cleanups, especially in DR setups. With expert support, the client restored full replication by applying Cassandra’s built-in maintenance tools and ensuring a safe node recovery process. The case also reinforced best practices around Cassandra disk thresholds and cluster monitoring. Moving forward, the client plans to add storage capacity and implement more frequent maintenance checks to prevent similar issues.