Case Studies

Problem: The client is preparing for an iPhone launch event, anticipating traffic spikes up to 200%. They require proactive monitoring of their Apache Cassandra production system during specified timeframes, with an upgrade to Severity 1 for immediate response during those periods. Process: Ticket Acknowledgment: We confirmed availability for the requested support dates and asked for […]

Problem: The client reported two main issues: One of the Kubernetes master nodes was in a “not ready” state. They needed to upgrade their Kubernetes version from 1.26 to 1.29. The client requested support to address these concerns. The client had already shut down the master node and was awaiting further instructions for troubleshooting. Process: […]

Problem: The client reported encountering a request timeout error when querying the PLDT Cassandra database in a production environment. The specific query involved selecting records from the jesi.service_monitoring table, which was attached along with a screenshot for further context. Process: Upon receiving the issue, the support team initiated an investigation. They first inquired about the […]

Problem: The client wanted to perform a rolling upgrade of the underlying operating system from RHEL 7 to RHEL 9 for their ETCD nodes in a Patroni-managed PostgreSQL cluster. The cluster contained three ETCD nodes and three Patroni-managed PostgreSQL instances (one primary and two standby). With a Recovery Point Objective (RPO) and Recovery Time Objective […]

Problem: The customer is facing frequent Docker Swarm re-elections, triggered even by brief server issues lasting just a few seconds. They are seeking guidance on how to modify the Swarm election timeout and whether adjusting this value will have any impact on the system. Process: Step 1: Initial Investigation The customer reported frequent leader re-elections […]

Problem: The client encountered a significant issue with their PostgreSQL database (PGDB). They reported that the filesystem (FS) utilization suddenly increased from 74% to 94% without any new objects being created. Despite their efforts to recreate the replication slot and restart PGPool, the filesystem remained at 94%. Logs revealed a termination error related to another […]

Problem: In the production environment of a multi-node OpenSearch cluster, the nodes frequently crashed due to Out-of-Memory (OOM) errors. Initially, the heap size was increased from 16 GB to 30 GB based on IBM’s recommendations, but the problem persisted. IBM further suggested increasing the number of shards from 16 to 64 to mitigate memory overload. […]

Problem: The client reported a replication issue in their A1 BG Production environment, consisting of a Patroni cluster with two PostgreSQL instances (Leader and Replica). Replication stopped, causing the leader’s /pgcluster file system to fill up with pg_wal files, leading to a full disk. The client requested help to identify the root cause of the […]

Problem: Following an upgrade from Cassandra 4.0.9 to 4.1.3, the client reported a noticeable increase in CPU utilization. The average CPU usage on their systems jumped from around 20% to approximately 37%. This escalation in CPU usage adversely impacted system performance and stability. The issue was notably more severe on servers running Red Hat Enterprise […]