Case Studies

Problem: In the Cassandra cluster, the database generated a large number of commit logs and didn’t delete them. Due to this, the commit logs filesystem is getting full and the database is crashing. This is relevant for all nodes. Process: Step 1: Initial Investigation and gather information from the client Initial troubleshooting and information gathering […]

Problem: The user was unable to reach the application page and received the error ‘Unauthenticated Page’. Process: Step 1: Initial Investigation The security issue pertains to an unauthenticated page within the Airflow version 2.5.2 instance. This unauthenticated page poses a potential security risk, as it can be accessed without proper authentication, potentially exposing sensitive information […]

Problem: The client reported an intermittent issue with their PostgreSQL database managed by Patroni. The error message encountered was “SSL SYSCALL error: EOF detected”. Despite checking the PostgreSQL logs and HAProxy logs, no corresponding errors were found. The client attempted to change the idle_in_transaction_session_timeout parameter from 1 hour to unlimited, but the error persisted. Solution: […]

Problem: The client has a two-datacenter (DC1 and DR1) Cassandra cluster. They encountered a failure while running nodetool repair on a node in DC1, which was traced to data corruption on a node in DR1. The logs indicated a corruption error in a specific SSTable file. Solution: Step 1. Initial Diagnosis: Ran nodetool repair in […]

Problem: The client was operating Jenkins 2.344 on Apache Tomcat 8.5.41 and required redirection from port 8084 (HTTP) to port 8443 (HTTPS). Although the “server.xml” and “web.xml” files were configured in the $CATALINA_HOME/conf/ directory, leading to successful redirection from http://jenkins:8084 to https://jenkins:8443, accessing http://jenkins:8084/jenkins (the application) did not redirect to port 8443. The cancellation of […]

Problem: The client’s production Apache Cassandra cluster experienced sudden native transport failure, leading to significant operational impact. Despite efforts to diagnose the problem using system logs and debug logs, the root cause remained unidentified. Native transport errors, particularly SSLPeerUnverifiedException, were prevalent in the debug logs, indicating authentication failures for multiple nodes in the cluster. Process: […]

Problem: The client, deploying the Prometheus operator using a community helm chart, encountered a security concern regarding the permissions granted to the Prometheus operator. Upon closer examination, it was discovered that the community helm chart provided overly permissive access rights, particularly with ‘*’ permissions for secrets and configmaps, as well as delete permissions for default […]

Problem: The client encountered a recurring issue within the Docker Swarm environment, wherein containers sporadically crashed with exit code 137. This behavior, indicative of potential memory-related issues, was exacerbated by the absence of corresponding container logs, complicating the diagnostic process. Process: Initial Inquiry and Investigation: Prompted by the client’s request for a Root Cause Analysis […]

Problem: The client is experiencing daily automatic restarts of their Cassandra database, potentially causing issues with application connectivity. Additionally, they’re encountering errors related to JVM memory, degraded mode, connection resets by peer, and null pointer exceptions on indexes. Solution: After a thorough analysis of the provided logs, the following findings and recommendations were made: Identify […]