Case Studies

Problem: The client’s operations team reported issues with triggering jobs via Apache Airflow, specifically through a custom solution, the dag_factory. While jobs triggered outside of the dag_factory worked without problems, those initiated through it were not being processed as expected. Attempts to gather logs in the Airflow UI yielded no entries, as the DAG triggering […]

Problem: The client reported a critical issue with the Cassandra cluster after adding a new data center and a rack containing three nodes. Despite bringing the new data center online, no data was being transferred from the source data center. Additionally, attempts to run a repair operation on the nodes were unsuccessful, which prevented the […]

Problem: The client needed to add a new data center to their existing Cassandra DB cluster for a critical project. However, upon starting Cassandra on the new server, it encountered a shutdown error due to a required node being offline. The error message, “A node required to move the data consistently is down,” indicated an […]

Problem: The client encountered issues with the data-explorer view functionality in their CKAN environment. While resources could be downloaded manually, the data-explorer view was unable to load. During the initial investigation, it was found that while the “datastore” plugin was enabled in the ckan.ini file, the ckan.datastore.write_url and ckan.datastore.read_url were not configured. The client was […]

Problem: The client is encountering a “SSTable corruption” issue when starting Cassandra in a new PLAB environment created using a CloudFormation template. After copying EBS volumes from a disaster recovery (DR) environment and making necessary adjustments in the cassandra.yaml file, they receive a series of NullPointerExceptions related to the SSTableReader while attempting to open SSTables. […]

Problem: The client experienced an authentication failure during health checks in their Pgpool-II setup, which led to a failover event. Despite updating the password in pool_passwd and pgpool.conf using the pg_md5 utility, the client continued to face the same issue. They observed that after failing over the node due to the authentication issue, they successfully […]

Problem: The client is preparing for an iPhone launch event, anticipating traffic spikes up to 200%. They require proactive monitoring of their Apache Cassandra production system during specified timeframes, with an upgrade to Severity 1 for immediate response during those periods. Process: Ticket Acknowledgment: We confirmed availability for the requested support dates and asked for […]

Problem: The client reported two main issues: One of the Kubernetes master nodes was in a “not ready” state. They needed to upgrade their Kubernetes version from 1.26 to 1.29. The client requested support to address these concerns. The client had already shut down the master node and was awaiting further instructions for troubleshooting. Process: […]

Problem: The client reported encountering a request timeout error when querying the PLDT Cassandra database in a production environment. The specific query involved selecting records from the jesi.service_monitoring table, which was attached along with a screenshot for further context. Process: Upon receiving the issue, the support team initiated an investigation. They first inquired about the […]