Database

Problem: The customer experienced issues with their Cassandra database, specifically with write failures and slow performance during nodetool repair operations. These issues were affecting the application’s ability to interact with the database, resulting in delays and failure to write data. The Cassandra cluster, consisting of 3 nodes in each of two data centers (US East […]

Problem: The client reported an issue where Apache Cassandra nodes in their multi-datacenter cluster were logging frequent errors related to SSL certificate validation, with entries like: DEBUG [Native-Transport-Requests-1] 2025-01-23 08:54:27,794 ServerConnection.java:140 - Failed to get peer certificates for peer /10.110.151.78:36376 javax.net.ssl.SSLPeerUnverifiedException: peer not verified Despite these log entries, the cluster continued to function normally, but […]

Problem: The client reported high memory usage on a production Apache Cassandra node, accompanied by frequent errors related to the ThreadPoolExecutor shutting down. This led to instability in the Cassandra service, including errors like java.util.concurrent.RejectedExecutionException, and resulted in a failure to execute repairs. Process: Step 1: Initial Identification The error logs provided by the client […]

Problem: The client needed to upgrade their production Apache Cassandra system from version 3.11.7 to 4.1.0 to take advantage of new features and improvements. They requested guidance on upgrade procedures and a reliable source for downloading the required RPM. Zero downtime was a critical requirement to ensure uninterrupted operations during the upgrade process. Process: Step […]

Problem: The client was experiencing performance issues with a self-managed Cassandra database cluster hosted on Azure VMs. A recent surge in data traffic led to high CPU utilization, causing significant system slowness and increased latency. The environment utilized SSDs for storage, but earlier attempts at recommended SSD optimizations yielded no significant improvement. In light of […]

Problem: The client planned to migrate a 5-node Cassandra cluster from an on-premises environment (version 3.11.8) to AWS (target version 4.1.5). The client requested guidance on the best migration strategy that ensures no downtime. Additionally, the client requested information on backup and restore procedures for the migration. Solution: The expert recommended a step-by-step approach. First […]

Problem: The client experienced issues with uneven data distribution across nodes after adding new nodes to an existing Cassandra cluster. Upon reviewing the “nodetool status” output, it was observed that new nodes were not receiving equal data compared to existing ones, resulting in significant data discrepancies between nodes. The client sought assistance in understanding why […]

Problem: The client reported a critical issue with the Cassandra cluster after adding a new data center and a rack containing three nodes. Despite bringing the new data center online, no data was being transferred from the source data center. Additionally, attempts to run a repair operation on the nodes were unsuccessful, which prevented the […]

Problem: The client needed to add a new data center to their existing Cassandra DB cluster for a critical project. However, upon starting Cassandra on the new server, it encountered a shutdown error due to a required node being offline. The error message, “A node required to move the data consistently is down,” indicated an […]