Database

Problem: A client from the telecommunications sector encountered critical issues while attempting to deploy a new Apache Cassandra cluster on AWS. After unpacking the Cassandra tarball and attempting to start the service manually, the Cassandra process failed to initialize due to a Java memory allocation error. The error indicated that the Java Virtual Machine (JVM) […]

Problem: The client reported that they had deployed a billing system for one of their customers, which uses PostgreSQL 15 within a high availability cluster managed by Patroni. As the system entered production, the customer requested that sensitive data within the database tables (such as names, addresses, phone numbers, and card information) be encrypted or […]

Problem: The client experienced a failover event in their PostgreSQL cluster managed by Patroni, between 01:00 and 02:00 AM. Process: Step 1: Initial Investigation Initial logs from PostgreSQL (postgresql-Fri-00.log and postgresql-Fri-01.log) revealed regular query activity. This included frequent queries from monitoring tools (pg_stat_all_tables, pg_locks, etc.), checkpoint logging, and client connection messages. There were no immediate […]

Problem: The client encountered a persistent issue related to file descriptor exhaustion on their PostgreSQL version 15 database, running on a Patroni High Availability cluster with RHEL 8.10. The PostgreSQL logs frequently reported the error: “out of file descriptors: Too many open files; release and retry” during database operations. Although the client had significantly increased […]

Problem: The client implemented Patroni High Availability PostgreSQL clusters using PostgreSQL version 15. The current configuration consisted of two-node clusters: one primary node and one replica node, with asynchronous replication between them. The client requested a more durable replication setup that would be resilient to operating system crashes, while maintaining high database performance, as the […]

Problem: The client was operating a 5-node Apache Cassandra cluster (version 4.1.5) and needed to establish secure access to the database from both an SRE server and an EC2 server. While basic connectivity (e.g., telnet) between the source and Cassandra target nodes was verified, direct access to Cassandra using cqlsh was unsuccessful. The client sought […]

Problem: The client encountered a failure while attempting to run a Cassandra backup using Commvault on their QAT cluster. The backup process failed with a 500 HTTP error originating from the local Priam REST endpoint: HTTP ERROR 500 Problem accessing /REST/v1/cassadmin/info. Reason: Commvault support traced the issue to the Priam service and advised the client […]

Problem: The client faced performance challenges while running nodetool cleanup on an Apache Cassandra 4.1.5 cluster during a node addition activity in a production environment. Specifically, the cleanup process was taking an unexpectedly long time on nodes with over 600GiB of data load, raising concerns about the overall timeline and impact on production workflows. The […]

Problem: The client reported high CPU utilization (up to 90%) across all nodes in their 3-node Apache Cassandra 4.1.3 cluster during full or incremental repair operations initiated from any single node. Despite relatively low data volumes (~25 GB per node), the CPU spike raised concerns about system performance, stability, and potential downtime during repairs. Process: […]