Case Studies

Problem: The client wanted to understand the logic behind the creation of SSTable (sorted string table) folders in Cassandra when data is written to a database table within a keyspace. Specifically, the question was about why each table has two directories in certain scenarios and how this logic affects the ability to achieve data backups […]

Problem: During a diagnostic call, it was observed that nodetool compactionstats was displaying 3,493 pending tasks related to the skyrt_prod1.event_state table. Despite new tasks being initiated and processed, these existing pending tasks remained uncleared, with no tasks currently in progress. This issue raised concerns about potential underlying problems with Cassandra’s compaction process. Solution: The accumulation […]

Problem: The client faced Spark issues during job submissions through Airflow. Out of 15 max DAGs that got triggered, there were always 3-4 dags getting triggered in the Initial state and never moving to the running state. A few DAGs also got stuck in the ContainerCreate state and never came up and ran. Process: Step […]

Problem: The client seeks guidance on passing multiple configuration files to Prometheus Adapter (P8s) version v0.11.1 without encountering size limitations or performance issues. They aim to aggregate custom metrics efficiently across microservices in Kubernetes. Process: The experts discussed various aspects of configuring Prometheus Adapter, addressing concerns such as size limitations with config maps and optimizing […]

Problem: The client, operating a Kafka Streams application version 3.3.1, encountered significant issues with managing changelog topics. Despite configuring the application for automatic record cleanup, the changelog topics exhibited unchecked growth. This accumulation of records risked deteriorating system performance and stability, potentially impacting the application’s reliability. Process: Initial Assessment: Issue Reporting: The client highlighted problems […]

Problem: The client reported that data storage space was being exhausted because the incremental backups were being saved in the same location as the data storage. This issue forced the client to delete incremental backups to free up space for new data. The client requested a consultation on how to configure incremental backups to be […]

Problem: The client reported several security vulnerabilities in Airflow version 2.5.0, including weak password policies such as allowing passwords with less than 8 characters, lack of password expiration, and absence of enforced password changes during the first login. These weaknesses compromise overall system security and user account integrity. Solution: To address these issues, the expert […]

Problem: The client reported encountering a recurring issue when attempting to execute scripts within a Docker container. The client consistently received an error code 137, indicating an Out of Memory (OOM) condition. Despite attempts to resolve the issue by restarting and reinstalling Docker, the problem persisted. Process: Gathering System Information: The Docker version being used; […]

Problem: The client requested to review cluster configuration and advise any changes to the configuration parameters to avoid any potential issues proactively. Additionally, the client requested advice on how to identify Cassandra database cluster’s workload. Process: 1) Data Collection: Gathered configuration files Collected the last 5,000 lines of the system log 2) Expert Review: Conducted […]