Case Studies

Problem: Triggering DAGs from Dag_factory for the initial load was not successful. The Dag_factory indicates success, but Collectors did not trigger. As a workaround, when the client triggers the DAG directly from the SQL collector, the drivers come up and are running. Solution: In a series of meetings, the team addressed Airflow issues encompassing the […]

Problem: The key challenge emerged when the university aimed to implement Kerberos Single Sign-On (SSO) for FreeIPA and configure Keycloak to seamlessly connect with FreeIPA. Two significant updates revealed obstacles: an inability to access a third-party application for Kerberos installation and the absence of a topology or visual representation of the configuration. Additionally, the university […]

Problem: The client reported encountering memory-related issues and tombstone cell messages in the system.log file of their Apache Cassandra deployment. Notable log entries included warnings about maximum memory usage, tombstone cells, and concerns about a hanging repair process. The issue seemed to have improved temporarily after increasing the heap size but resurfaced after two days. […]

Problem: The challenge arises when attempting to upgrade from Spring Boot (SB) version 2.7.x to SB 3.1.x due to compatibility issues with OpenJPA. No compatible version of OpenJPA has been found for Spring Boot 3.1.x. Upgrading to Spring Boot 3.1.x presents hurdles in terms of OpenJPA compatibility. The current version of OpenJPA lacks support for […]

Problem: The problem is that commit logs in the production Cassandra cluster are accumulating excessively without being deleted, leading to a full filesystem and subsequent database crashes. Process: Step 1: Hardware Specifications and Disk Space: Requested hardware specifications for each Cassandra node. Checked disk space on all nodes using df -h command. Step 2: System […]

Problem: The client encountered a critical issue where LW Locking on the PostgreSQL DB level, specifically the MultiXactOffsetControlLock, led to a complete outage of the production system. This resulted in 255 sessions being locked for 4 hours causing a significant impact on the system’s availability. Solution: PostgreSQL Version and Configuration: The PostgreSQL version in use […]

Problem: The client encountered challenges connecting to the cqlsh on several nodes within their Cassandra cluster. Additionally, discrepancies were noted in the output of “nodetool status” across different nodes, with certain nodes appearing as down. Seeking assistance, the client provided output files for analysis, prompting intervention to rectify the connectivity issues. Process: Check Network Connectivity: […]

Problem: The client encountered an error during the installation of a Helm chart, specifically related to the NGINX admission controller. The installation failed due to a validation error in NGINX Ingress, indicating a synchronization issue with the ingress event handlers. This problem arose when multiple ingresses used the same secret, causing complications with secret refreshing […]

Introduction The installation of Airflow and Apache Spark can be fine-tuned for optimalperformance by adjusting over 150 environment variables, thereby maximizingthe number of DAGs running and fully utilizing allocated resources.During recent hosted support sessions with an ISV that develops software fortelecommunications companies, we encountered multiple challenges ineffectively scaling their Airflow deployment. Specifically, we focused onoptimizing […]