Database

Problem: The client needed to change the passwords of all users in the Cassandra cluster. It was specifically inquired about the necessity of changing the default password for the “cassandra” superuser and requested a step-by-step guide, along with precautions to prevent any impact on the application. Process: The expert provided a detailed response with the […]

Problem: The table is not updated immediately in Cassandra DB. The reference id table is not updated sometimes and the application gets old values. Process: Step 1: In the initial investigation and troubleshooting of the DB our experts asked the client the following questions: Cluster Configuration How many nodes are there in the cluster? What […]

Problem: The client experienced frequent warnings in their Cassandra nodes, indicated by the log entry: INFO [CompactionExecutor:37] 2023-02-23 12:00:12,268 NoSpamLogger.java:91 – Maximum memory usage reached (536870912), cannot allocate chunk of 1048576 The client requested an investigation into the potential impact of increasing the file_cache_size_in_mb in the cassandra.yaml file, whether a restart (bounce) would be necessary, […]

Problem: The client encountered a Cassandra exception: “WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM.” This issue, occurring since March 9, 2023, revolves around an INSERT INTO query. They seek troubleshooting assistance as this exception had not occurred before that date. Process: Step 1 – Initial Investigation and Troubleshooting: The expert team initiated […]

Problem: The client reported issues with Keepalived crashes leading to high availability disruptions and application errors, particularly connection timeouts with the PostgreSQL server. Initial investigations revealed suspicions of network instability and outdated software versions contributing to the problem. Process: Requesting initial information for further investigation of the problem The number of servers in the HAProxy […]

Problem: In the Cassandra cluster, the database generated a large number of commit logs and didn’t delete them. Due to this, the commit logs filesystem is getting full and the database is crashing. This is relevant for all nodes. Process: Step 1: Initial Investigation and gather information from the client Initial troubleshooting and information gathering […]

Problem: The client reported an intermittent issue with their PostgreSQL database managed by Patroni. The error message encountered was “SSL SYSCALL error: EOF detected”. Despite checking the PostgreSQL logs and HAProxy logs, no corresponding errors were found. The client attempted to change the idle_in_transaction_session_timeout parameter from 1 hour to unlimited, but the error persisted. Solution: […]

Problem: The client has a two-datacenter (DC1 and DR1) Cassandra cluster. They encountered a failure while running nodetool repair on a node in DC1, which was traced to data corruption on a node in DR1. The logs indicated a corruption error in a specific SSTable file. Solution: Step 1. Initial Diagnosis: Ran nodetool repair in […]

Problem: The client’s production Apache Cassandra cluster experienced sudden native transport failure, leading to significant operational impact. Despite efforts to diagnose the problem using system logs and debug logs, the root cause remained unidentified. Native transport errors, particularly SSLPeerUnverifiedException, were prevalent in the debug logs, indicating authentication failures for multiple nodes in the cluster. Process: […]