Case Studies

Problem: The client encountered an error while updating the Prometheus adapter in the Azure environment, specifically related to the prometheus-adapter-configmap. The error stemmed from the generated configmap exceeding the Kubernetes limit, preventing successful deployment. The client sought guidance on efficiently splitting the configuration for deployment across multiple namespaces. Solution: Step 1: Environment Preparation: Experts established […]

Problem: The Production (Prod) and Disaster Recovery (DR) environments for Cassandra databases exhibit different sizes, as observed from the nodetool status output. The client needs to identify the reason behind this discrepancy and ensure the readiness and reliability of the Disaster Recovery environment for use in the event of a failure. Process: The client provided […]

Problem: The issue revolves around an unexpected change in the TTL for our Cassandra cluster. There are approximately 28 tables, six of which contain around 2 million records each. The requirement is to update the TTL for these tables to three years. However, TTL is configured at the row level. The client seeks alternative methods […]

Problem: The client reported issues with all dags triggered from dag-factory failing, with associated problems such as pods not coming up when triggered and errors in collector pods. Furthermore, there was a lack of error messages in the airflow pods logs, making it challenging to identify the root cause. Process: In order to address the […]

Problem: The client sought consultation on the best way establishing the standard Cassandra topology for a production-ready environment. Process: Cassandra Configurations: 2 Data Centers (PROD and DR). Each DC has 3 nodes of Cassandra. 2 DC are part of one cluster, thereby forming a ring architecture of 6 nodes (3 PROD + 3 DR). RF […]

Problem: The client is experiencing failures during the copy/export process, resulting in Cassandra crashes and errors related to memory exhaustion. These failures occur due to incorrectly formatted date fields and memory-intensive processes involved in copy/export operations. Solution: Identification of Date Formatting Issue: The expert identified that the failures were caused by incorrectly formatted date fields […]

Problem: The problem is that commit logs in the production Cassandra cluster are accumulating excessively without being deleted, leading to a full filesystem and subsequent database crashes. Process: FreeIPA-Replication-Troubleshooting We have tried to reproduce the setup by performing the following actions. Installing FreeIPA Server on CentOS 7 server. Which had the default version of FreeIPA […]

Problem: There are two problems with Prometheus monitoring setup: the retention setting isn’t functioning correctly, leading to excessive data storage, and the Prometheus database is frequently getting corrupted, despite varying levels of workload and resource allocation across different environments. Process: Case Details: Environment 1: Approximately 23GB of data is generated in Prometheus across a 3-hour […]

Problem: During a migration from Kafka Streams 2.5.0 to 3.3.1, a client encounters a “Producer Fenced Exception” while processing aggregation tasks using a Processor in Kafka Streams. The issue occurs when attempting to update a State Store changelog topic, initiated by operations reading data from upstream and processing it with a punctuator. Process: Questions: Should […]