Case Studies

Problem: The application fails to establish an SSL connection to the database, displaying the error “could not accept SSL connection: Success” in the database logs. Despite this, manual `psql` connections using SSL work fine. Process: Step 1 – Verify Connection String: Confirm that the application’s connection string includes the necessary SSL parameters. Example: postgresql://user:password@hostname:port/dbname?sslmode=require Step […]

Problem: Zipkin version 2.24.2 does not support Elasticsearch 8.x, which poses a significant obstacle as many of our clusters are already upgraded to Elasticsearch 8.x. This compatibility issue needs to be addressed to ensure seamless operation of our monitoring and tracing functionalities. Solution: Assessment and Documentation: Conducted thorough analysis and documented the current compatibility status […]

Problem: The client experienced connection issues on the PostgreSQL database server, with an abnormally high number of connections reaching around 20,000 at a given time. The client asked for assistance from the expert team in identifying the possible reasons behind this issue. Based on the client’s analysis, there were multiple wait events on HAProxy, and […]

Problem: The client reported an issue where the Prometheus pod was crashing in the production environment. The error logs indicated a variety of issues including “Terminated Reason: Error” and messages about unhealthy blocks and existing lock files. The specific error message highlighted was: Last State: Terminated Reason: Error Message: und healthy block” mint=1680069600000 maxt=1680091200000 ulid=01GWPY8332RWJAPSFZ8KNAJQ9H […]

Problem: Restoring a Cassandra 3.11 snapshot to a 4.0.6 cluster using the nodetool refresh command results in an empty table, indicating a potential compatibility issue. This affects the DR environment, which needs to accurately replicate the PROD environment’s data. Solution: Step 1. Verify Snapshot Content: Use nodetool listsnapshots and a test environment to ensure the […]

Problem: The client ran Cassandra v4.0.6 in non-production environments and noticed that the “Cassandra” superuser’s password (which was changed two months ago) was observed to be reset to its old password (default password “Cassandra”). After patches were applied to the OS the server rebooted (a monthly activity). The client didn’t see any evidence of someone […]

Problem: The client reported that their data storage space was being rapidly exhausted because incremental backups were stored in the same location as the primary data storage. This resulted in the storage becoming full, forcing them to delete incremental backups to make room for new data. The client requested guidance on configuring incremental backups to […]

Problem: Three instances of the production Postgres Database cluster experienced crashes with “segmentation fault” errors within 30 days. Despite no recent changes in the system, the issue persisted, prompting the need for investigation to identify the root cause. Process: Upon receiving the initial report of the issue, the experts engaged with the client to gather […]

Problem: The kube-api pod was frequently restarting with the error message: apiserver received an error that is not a metav1.Status: rpctypes.EtcdError{code:0xe, desc:”etcdserver: request timed out”}. This issue began following an etcd data migration from a single-node Kubernetes cluster to a multi-node cluster using an unconventional method. Despite multiple attempts to resolve the issue, the problem […]