Database

Problem: The client was operating a 5-node Apache Cassandra cluster (version 4.1.5) and needed to establish secure access to the database from both an SRE server and an EC2 server. While basic connectivity (e.g., telnet) between the source and Cassandra target nodes was verified, direct access to Cassandra using cqlsh was unsuccessful. The client sought […]

Problem: The client encountered a failure while attempting to run a Cassandra backup using Commvault on their QAT cluster. The backup process failed with a 500 HTTP error originating from the local Priam REST endpoint: HTTP ERROR 500 Problem accessing /REST/v1/cassadmin/info. Reason: Commvault support traced the issue to the Priam service and advised the client […]

Problem: The client faced performance challenges while running nodetool cleanup on an Apache Cassandra 4.1.5 cluster during a node addition activity in a production environment. Specifically, the cleanup process was taking an unexpectedly long time on nodes with over 600GiB of data load, raising concerns about the overall timeline and impact on production workflows. The […]

Problem: The client reported high CPU utilization (up to 90%) across all nodes in their 3-node Apache Cassandra 4.1.3 cluster during full or incremental repair operations initiated from any single node. Despite relatively low data volumes (~25 GB per node), the CPU spike raised concerns about system performance, stability, and potential downtime during repairs. Process: […]

Problem: The client reported a sudden and significant drop in Apache Cassandra performance on a 4-node cluster. The issue appeared without any recent configuration or infrastructure changes. The application started experiencing frequent timeouts, and restarting Cassandra services on all nodes did not resolve the problem. The client provided various monitoring graphs, system logs, and other […]

Problem: The client needed to scale their production Cassandra cluster from 6 nodes to 12 nodes (3 to 6 nodes per data center) without any downtime. Their existing setup includes Cassandra version 4.1.6, with two data centers (PROD and DR), each containing 3 nodes, forming a 6-node cluster with a replication factor of 3 and […]

Problem: The client encountered a critical issue while starting one of their production pods during a Cassandra migration. Although the PostgreSQL migration completed successfully, the Cassandra migration failed with a com.datastax.oss.driver.api.core.AllNodesFailedException, indicating that the driver could not connect to any Cassandra nodes. This blocked the production deployment. Process: Step 1 – Initial Analysis The logs […]

Problem: The client was using Apache Cassandra 4.1.5 installed via a tarball extraction on an AWS EC2 machine and wanted to upgrade both their Cassandra version and the operating system. The installation was done manually using the tarball method, and the client needed to understand the feasibility and potential challenges involved in upgrading the OS […]

Problem: The client experienced a failure in running repairs in Apache Cassandra due to corruption in hint files. Additionally, a node in the cluster went down and could not be brought back up, causing concerns about data consistency and cluster stability. Process: Step 1: Initial Investigation The client observed errors related to corrupted hint files, […]