Database

Problem: The client sought consultation on the best way establishing the standard Cassandra topology for a production-ready environment. Process: Cassandra Configurations: 2 Data Centers (PROD and DR). Each DC has 3 nodes of Cassandra. 2 DC are part of one cluster, thereby forming a ring architecture of 6 nodes (3 PROD + 3 DR). RF […]

Problem: The client is experiencing failures during the copy/export process, resulting in Cassandra crashes and errors related to memory exhaustion. These failures occur due to incorrectly formatted date fields and memory-intensive processes involved in copy/export operations. Solution: Identification of Date Formatting Issue: The expert identified that the failures were caused by incorrectly formatted date fields […]

Problem: The client reported encountering memory-related issues and tombstone cell messages in the system.log file of their Apache Cassandra deployment. Notable log entries included warnings about maximum memory usage, tombstone cells, and concerns about a hanging repair process. The issue seemed to have improved temporarily after increasing the heap size but resurfaced after two days. […]

Problem: The problem is that commit logs in the production Cassandra cluster are accumulating excessively without being deleted, leading to a full filesystem and subsequent database crashes. Process: Step 1: Hardware Specifications and Disk Space: Requested hardware specifications for each Cassandra node. Checked disk space on all nodes using df -h command. Step 2: System […]

Problem: The client encountered a critical issue where LW Locking on the PostgreSQL DB level, specifically the MultiXactOffsetControlLock, led to a complete outage of the production system. This resulted in 255 sessions being locked for 4 hours causing a significant impact on the system’s availability. Solution: PostgreSQL Version and Configuration: The PostgreSQL version in use […]

Problem: The client encountered challenges connecting to the cqlsh on several nodes within their Cassandra cluster. Additionally, discrepancies were noted in the output of “nodetool status” across different nodes, with certain nodes appearing as down. Seeking assistance, the client provided output files for analysis, prompting intervention to rectify the connectivity issues. Process: Check Network Connectivity: […]

Problem: The client reported high memory consumption on both leader (Node 1) and replica (Node 2) nodes in PostgreSQL version 13. Memory utilization on both nodes was observed to be significantly elevated. On Node 1, high memory usage was associated with PostgreSQL processes such as checkpoint and background writer operations, while Node 2 was undergoing […]

Problem: The client reported that Cassandra 2.2.5, running in a single-node configuration, crashed and failed to start. The error logs pointed to a “CorruptSSTableException” in the “system_traces.events” table, indicating corruption in an SSTable file. Since this was a critical system table, the client needed a way to bypass the corrupted data and restart the Cassandra […]

Problem: The client reported a critical increase in pending tasks on one of the nodes within their Cassandra cluster. This issue was causing concern, and the client sought assistance in understanding the root cause and implementing a resolution. Process: The client initially executed the nodetool compaction-stats -H command on the affected node and restarted it, […]