Case Studies

Problem: A client managing a multi-datacenter Apache Cassandra setup observed an inconsistency in rack configuration between two datacenters. In the primary datacenter (`dc1`), all three nodes were configured with the same rack identifier (`rack1`), whereas in the secondary datacenter (`drdc`), each node was correctly configured with unique rack values (`rack1`, `rack2`, `rack3`). This raised concerns […]

Problem: The client encountered a critical issue in their Apache Cassandra setup: the Disaster Recovery (DR) cluster was consuming significantly more filesystem (FS) space than the primary cluster, despite having the same data replication strategy. As a result, one of the DR nodes hit 100% disk usage, causing the replica to go down. The client […]

Problem: The client wanted to integrate Apache Cassandra metrics into Prometheus for monitoring and alerting. While using JMX Exporter with Node Exporter, they noticed that too many unwanted metrics were being fetched. This resulted in performance issues such as delayed response times from the curl command and unnecessary load on Prometheus. Additionally, key metrics like […]

Problem: A client from the telecommunications sector encountered critical issues while attempting to deploy a new Apache Cassandra cluster on AWS. After unpacking the Cassandra tarball and attempting to start the service manually, the Cassandra process failed to initialize due to a Java memory allocation error. The error indicated that the Java Virtual Machine (JVM) […]

Problem: The client reported that they had deployed a billing system for one of their customers, which uses PostgreSQL 15 within a high availability cluster managed by Patroni. As the system entered production, the customer requested that sensitive data within the database tables (such as names, addresses, phone numbers, and card information) be encrypted or […]

Problem: The client experienced a failover event in their PostgreSQL cluster managed by Patroni, between 01:00 and 02:00 AM. Process: Step 1: Initial Investigation Initial logs from PostgreSQL (postgresql-Fri-00.log and postgresql-Fri-01.log) revealed regular query activity. This included frequent queries from monitoring tools (pg_stat_all_tables, pg_locks, etc.), checkpoint logging, and client connection messages. There were no immediate […]

Problem: The client encountered a persistent issue related to file descriptor exhaustion on their PostgreSQL version 15 database, running on a Patroni High Availability cluster with RHEL 8.10. The PostgreSQL logs frequently reported the error: “out of file descriptors: Too many open files; release and retry” during database operations. Although the client had significantly increased […]

Problem: The client implemented Patroni High Availability PostgreSQL clusters using PostgreSQL version 15. The current configuration consisted of two-node clusters: one primary node and one replica node, with asynchronous replication between them. The client requested a more durable replication setup that would be resilient to operating system crashes, while maintaining high database performance, as the […]

Problem: A financial services client encountered critical SSL-related errors while deploying a two-node OpenSearch 1.3.6 cluster for high availability. Despite both nodes appearing operational, accessing indices or interacting with the cluster through a Java application resulted in errors such as: SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment CertPathBuilderException: unable to find valid certification path […]