Case Studies

Problem: The client encountered an issue with desynchronization of nodes in an ETCD cluster running on RHEL 8.8. Error logs indicated significant disk write delays (slow fdatasync), which caused the Patroni cluster to fail and become unavailable. Process: Step 1 – Initial Analysis The expert asked the client to check the health of the cluster […]

Problem: The client encountered frequent master (manager) re-elections in their production Docker Swarm cluster, despite having the dispatcher-heartbeat value set to 2 minutes. These re-elections were happening within fractions of a second, causing concerns around Swarm stability and service availability. The client’s Docker environment was based on version 1.13.1 running on RHEL 7.9. Key symptoms […]

Problem: The client faced recurring Kafka sink connector failures (e.g., chf-cdr-sftp-sink-connector) in a Kubernetes environment (Kafka 3.2.0 with three brokers and ZooKeeper). The failures were caused by corrupt messages at specific offsets, leading to task crashes. Despite skipping corrupt offsets and restarting connectors, the issue persisted, requiring a more permanent solution. Process: Step 1: Environment […]

Problem: The client faced an issue integrating Jenkins with Keycloak for authentication. While the Jenkins UI successfully authenticated users via Keycloak, API calls from backend services were failing. According to Jenkins’ documentation, API requests should be authenticated using an API token, but despite following the recommended steps, the client encountered authentication failures (403 Forbidden & […]

Problem: The client experienced issues with Docker Swarm configuration in production. Specifically, when a container restarted, the application failed to recover properly. The client requested a review of the configuration to identify the root cause and potential improvements to enhance the cluster’s functionality. Process: Step 1: Initial Investigation The client provided details of the Docker […]

Problem: The client was using Apache Cassandra 4.1.5 installed via a tarball extraction on an AWS EC2 machine and wanted to upgrade both their Cassandra version and the operating system. The installation was done manually using the tarball method, and the client needed to understand the feasibility and potential challenges involved in upgrading the OS […]

Problem: The client experienced a failure in running repairs in Apache Cassandra due to corruption in hint files. Additionally, a node in the cluster went down and could not be brought back up, causing concerns about data consistency and cluster stability. Process: Step 1: Initial Investigation The client observed errors related to corrupted hint files, […]

Problem: The client experienced a problem with query slowness in their PostgreSQL database. Several queries were running slowly, and the application became unresponsive during the issue. The client required assistance in diagnosing and optimizing the queries contributing to the performance issues. Process: Step 1 – Initial Investigation The expert reviewed the PGAWR reports for the […]

Problem: The client encountered a high availability issue in their Cassandra cluster, consisting of five nodes deployed on AWS EC2. After shutting down two servers (10.51.44.25 and 10.51.46.144), it became impossible to connect to the database, even though the other nodes remained online. The issue manifested as an authentication error when trying to connect to […]