Case Studies

Problem: The client reported an issue where the Prometheus pod was crashing in the production environment. The error logs indicated a variety of issues including “Terminated Reason: Error” and messages about unhealthy blocks and existing lock files. The specific error message highlighted was: Last State: Terminated Reason: Error Message: und healthy block” mint=1680069600000 maxt=1680091200000 ulid=01GWPY8332RWJAPSFZ8KNAJQ9H […]

Problem: Restoring a Cassandra 3.11 snapshot to a 4.0.6 cluster using the nodetool refresh command results in an empty table, indicating a potential compatibility issue. This affects the DR environment, which needs to accurately replicate the PROD environment’s data. Solution: Step 1. Verify Snapshot Content: Use nodetool listsnapshots and a test environment to ensure the […]

Problem: The client reported that their data storage space was being rapidly exhausted because incremental backups were stored in the same location as the primary data storage. This resulted in the storage becoming full, forcing them to delete incremental backups to make room for new data. The client requested guidance on configuring incremental backups to […]

Problem: The client ran Cassandra v4.0.6 in non-production environments and noticed that the “Cassandra” superuser’s password (which was changed two months ago) was observed to be reset to its old password (default password “Cassandra”). After patches were applied to the OS the server rebooted (a monthly activity). The client didn’t see any evidence of someone […]

Problem: Three instances of the production Postgres Database cluster experienced crashes with “segmentation fault” errors within 30 days. Despite no recent changes in the system, the issue persisted, prompting the need for investigation to identify the root cause. Process: Upon receiving the initial report of the issue, the experts engaged with the client to gather […]

Problem: The kube-api pod was frequently restarting with the error message: apiserver received an error that is not a metav1.Status: rpctypes.EtcdError{code:0xe, desc:”etcdserver: request timed out”}. This issue began following an etcd data migration from a single-node Kubernetes cluster to a multi-node cluster using an unconventional method. Despite multiple attempts to resolve the issue, the problem […]

Problem: PG Prod Node 1 failed over to Node 2, accompanied by high WAL (Write-Ahead Logging) generation. The client requested an investigation into the cause of the failover on Node 1. Process: To conduct a thorough investigation, the following data and logs were requested: PostgreSQL logs Cluster logs Database logs WAL logs Cluster configuration Monitoring […]

Problem: The client had a 10-node cluster across two data centers, with 5 nodes in each. They ran nodetool repair on one keyspace, but after 6 hours, it was stuck at 99% completion. They requested guidance on how to proceed with the issue, noting that their product version was ‘ReleaseVersion: 2.2.5.’ Process: Step 1 – […]

Problem: Cassandra v4.0.6 in non-production environments experienced an issue where the “cassandra” superuser password, which had been changed two months prior, reset to its default password (“cassandra”) after applying OS patches and rebooting the server. No manual password changes were evident in the audit logs. Solution: Initial Steps: The superuser “cassandra” initially had the default […]