Case Studies

Problem: Client reported an issue with Cassandra version 4 after upgrading from version 3. Running the “nodetool repair” command on a cluster with 2 data centers (3 nodes each) resulted in an error indicating that the incremental repair session failed. This issue did not occur with version 3, and all nodes showed no pending compaction […]

Problem: On July 8th, 2024, all Docker containers on all nodes within a Docker Swarm cluster suddenly crashed. The cluster consisted of 13 nodes: 1 master, 2 reachable, and 10 worker nodes. The initial logs indicated a problem with the RAFT consensus algorithm attempting and failing to elect a leader multiple times. Process: Upon receiving […]

Problem: The Jenkins server experienced significant performance issues characterized by excessive thread creation and inadequate resource allocation. Symptoms included system freezes, failures to execute commands, and frequent application errors related to memory and resource limits. Process: Initial Investigation: Error Identification: Logs and system monitoring revealed critical errors related to insufficient memory and resource limits. Key […]

Problem: Argo CD, version v2.10.4, deployed in a high-load environment with approximately 480 applications, encountered severe performance issues. Specifically, the refresh operations were inconsistent, sometimes taking up to 16 minutes, impacting deployment efficiency and causing significant CPU consumption by the controller pod. Despite efforts to adjust configuration parameters based on Argo CD documentation, the issue […]

Problem: Some client servers are not receiving updated data from our IPA servers. For example, listing hosts in a specific host group on one client server shows missing hosts: ~]# ipa host-find --in-hostgroups=rhel9_hosts | grep Host | grep -i ra Host name: india In contrast, the same command on another server shows additional hosts: ~]# […]

Problem: The client reported a critical issue with Solr 6.6.5, where Solr cores were becoming corrupt when new data was pushed through the Data Import Handler (DIH). This issue led to significant data loss and functionality impact, as existing indexed data became inaccessible, resulting in zero search results despite having content in Solr. Process: Step […]

Problem: After upgrading Jenkins to LTS Version 2.440.1 and updating installed plugins, the client encountered an issue with active choice parameters on the “build with parameters” page. Specifically, the page is waiting for full rendering before executing scripts within the active choice parameters, notably those involving HTTP requests. This delay has caused certain features to […]

Problem: Not Showing Patroni Output in PG DR Database. The team faced an issue where the Patroni status was visible on the production (PROD) server but not on the disaster recovery (DR) server. The discrepancy hindered the ability to monitor the DR environment effectively. Process: Request Patroni Configuration and Logs: Obtain patroni.yml from both PROD […]

Problem: The client was planning an upgrade of the MySQL database server from Red Hat Enterprise Linux (RHEL) 7.x to RHEL 8.x. The MySQL version currently running on the server was 14.14 (Distrib 5.7.34). The client needed to understand the compatibility of this MySQL version with RHEL 8.x and sought guidance on any additional steps […]