Database

Problem: In the Cassandra cluster, the database generated a large number of commit logs and didn’t delete them. Due to this, the commit logs filesystem is getting full and the database is crashing. This is relevant for all nodes. Process: Step 1: Initial Investigation and gather information from the client Initial troubleshooting and information gathering […]

Problem: The client reported an intermittent issue with their PostgreSQL database managed by Patroni. The error message encountered was “SSL SYSCALL error: EOF detected”. Despite checking the PostgreSQL logs and HAProxy logs, no corresponding errors were found. The client attempted to change the idle_in_transaction_session_timeout parameter from 1 hour to unlimited, but the error persisted. Solution: […]

Problem: The client has a two-datacenter (DC1 and DR1) Cassandra cluster. They encountered a failure while running nodetool repair on a node in DC1, which was traced to data corruption on a node in DR1. The logs indicated a corruption error in a specific SSTable file. Solution: Step 1. Initial Diagnosis: Ran nodetool repair in […]

Problem: The client’s production Apache Cassandra cluster experienced sudden native transport failure, leading to significant operational impact. Despite efforts to diagnose the problem using system logs and debug logs, the root cause remained unidentified. Native transport errors, particularly SSLPeerUnverifiedException, were prevalent in the debug logs, indicating authentication failures for multiple nodes in the cluster. Process: […]

Problem: The client is experiencing daily automatic restarts of their Cassandra database, potentially causing issues with application connectivity. Additionally, they’re encountering errors related to JVM memory, degraded mode, connection resets by peer, and null pointer exceptions on indexes. Solution: After a thorough analysis of the provided logs, the following findings and recommendations were made: Identify […]

Problem: The client reported instances of Cassandra databases experiencing unavailability and performance degradation, leading to application connection errors. Specifically, nodes were found to be unresponsive, and memory and CPU utilization spiked unexpectedly. Additionally, there were concerns about data inconsistencies between data centers and excessive tombstones affecting performance. Process: Initial Investigation: Upon initial investigation, it was […]

Problem: After deleting 52 million rows from a table, the client performed a VACUUM FULL and REINDEX on the table. Following these maintenance operations, the performance of large joins involving this table has significantly deteriorated, with query execution times increasing from minutes to hours. The task is to identify the cause of this performance regression […]

Problem: The client had 10 monthly bill cycles, and during such bill cycle days, the billing process was connected to the DB cluster in 20 streams. After the process had started the replication the above-mentioned billing process failed with the “could not send data to the client: Connection reset by peer” error. The client reduced […]

Problem: The client experienced timeout errors when running adhoc queries in an Apache Cassandra cluster. Specifically, a query with multiple conditions was timing out due to the coordinator node not receiving responses from replica nodes. Process: Here are the steps taken by the expert in the process of resolving the problem: Assessment of Query Efficiency: […]