Case Studies

Problem: The client’s production environment includes Rancher installed on two clusters: a Rancher cluster and an application cluster. During the cluster setup, the kubelet certificate was generated with a validity of one year, which recently expired. According to the Rancher RKE documentation, additional configuration is needed to manage certificate validity. The client observed inconsistencies: Some […]

Problem: The client needed to upgrade their production Apache Cassandra system from version 3.11.7 to 4.1.0 to take advantage of new features and improvements. They requested guidance on upgrade procedures and a reliable source for downloading the required RPM. Zero downtime was a critical requirement to ensure uninterrupted operations during the upgrade process. Process: Step […]

Problem: The client faced issues with frequent re-elections in a Docker Swarm cluster whenever there were brief server-level disruptions. They sought guidance on modifying the swarm election timeout to stabilize the cluster and prevent unnecessary re-elections. Additionally, they wanted to understand the relationship between election timeout, heartbeat, and dispatcher-heartbeat settings. Process: Step 1: Initial Investigation […]

Problem: The client, a company using FreeIPA for identity management, encountered issues when running the ipa-healthcheck command. The system was returning errors related to non-existent servers, which had been decommissioned as part of a recent infrastructure migration. These errors were causing the ipa-healthcheck command to fail and reported old servers that no longer existed in […]

Problem: The client was experiencing performance issues with a self-managed Cassandra database cluster hosted on Azure VMs. A recent surge in data traffic led to high CPU utilization, causing significant system slowness and increased latency. The environment utilized SSDs for storage, but earlier attempts at recommended SSD optimizations yielded no significant improvement. In light of […]

Problem: The client implemented a 4-node OpenSearch cluster to ensure high availability for their application. When all four nodes were operational, both indexing and searching worked seamlessly. However, during a high availability test where two nodes were intentionally turned off, the indexing process stalled, and no documents were processed. Indexing resumed only after the two […]

Problem: The client, a FinTech company, managing thousands of microservices using Istio in sidecar proxy mode, faced high CPU and memory utilization. This was caused by the overhead from Istio sidecars, which were handling: Traffic encryption and decryption with mTLS. Traffic routing (rate limiting, retries) and policy management. Telemetry generation for monitoring and tracing tools. […]

Problem: The client used Istio to manage service communication in a distributed microservices architecture. Centralized services, including Gitlab, Keycloak, Vault, and others, were hosted in an Amazon EKS cluster and accessed via a WireGuard-based VPN mesh (Netbird) from 10 external Kubernetes clusters. Despite having all services exposed through Istio ingress gateways, external clusters experienced frequent […]

Problem: The client reported delays in the readiness of ingress virtual services and difficulty accessing services through DNS names. Despite using Istio for service-to-service communication and centralized services like Keycloak, Gitlab, Vault, and others, the setup was taking too long, especially when resolving DNS names for these services. The delay was primarily due to Crossplane […]