Case Studies

Problem: The client encountered a Cassandra exception: “WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM.” This issue, occurring since March 9, 2023, revolves around an INSERT INTO query. They seek troubleshooting assistance as this exception had not occurred before that date. Process: Step 1 – Initial Investigation and Troubleshooting: The expert team initiated […]

Problem: The client reported issues with Keepalived crashes leading to high availability disruptions and application errors, particularly connection timeouts with the PostgreSQL server. Initial investigations revealed suspicions of network instability and outdated software versions contributing to the problem. Process: Requesting initial information for further investigation of the problem The number of servers in the HAProxy […]

Problem: In the Cassandra cluster, the database generated a large number of commit logs and didn’t delete them. Due to this, the commit logs filesystem is getting full and the database is crashing. This is relevant for all nodes. Process: Step 1: Initial Investigation and gather information from the client Initial troubleshooting and information gathering […]

Problem: The user was unable to reach the application page and received the error ‘Unauthenticated Page’. Process: Step 1: Initial Investigation The security issue pertains to an unauthenticated page within the Airflow version 2.5.2 instance. This unauthenticated page poses a potential security risk, as it can be accessed without proper authentication, potentially exposing sensitive information […]

Problem: The client reported an intermittent issue with their PostgreSQL database managed by Patroni. The error message encountered was “SSL SYSCALL error: EOF detected”. Despite checking the PostgreSQL logs and HAProxy logs, no corresponding errors were found. The client attempted to change the idle_in_transaction_session_timeout parameter from 1 hour to unlimited, but the error persisted. Solution: […]

Problem: The client has a two-datacenter (DC1 and DR1) Cassandra cluster. They encountered a failure while running nodetool repair on a node in DC1, which was traced to data corruption on a node in DR1. The logs indicated a corruption error in a specific SSTable file. Solution: Step 1. Initial Diagnosis: Ran nodetool repair in […]

Problem: The client was operating Jenkins 2.344 on Apache Tomcat 8.5.41 and required redirection from port 8084 (HTTP) to port 8443 (HTTPS). Although the “server.xml” and “web.xml” files were configured in the $CATALINA_HOME/conf/ directory, leading to successful redirection from http://jenkins:8084 to https://jenkins:8443, accessing http://jenkins:8084/jenkins (the application) did not redirect to port 8443. The cancellation of […]

Problem: The client’s production Apache Cassandra cluster experienced sudden native transport failure, leading to significant operational impact. Despite efforts to diagnose the problem using system logs and debug logs, the root cause remained unidentified. Native transport errors, particularly SSLPeerUnverifiedException, were prevalent in the debug logs, indicating authentication failures for multiple nodes in the cluster. Process: […]

Problem: The client, deploying the Prometheus operator using a community helm chart, encountered a security concern regarding the permissions granted to the Prometheus operator. Upon closer examination, it was discovered that the community helm chart provided overly permissive access rights, particularly with ‘*’ permissions for secrets and configmaps, as well as delete permissions for default […]