Case Studies

Problem: The kube-api pod was frequently restarting with the error message: apiserver received an error that is not a metav1.Status: rpctypes.EtcdError{code:0xe, desc:”etcdserver: request timed out”}. This issue began following an etcd data migration from a single-node Kubernetes cluster to a multi-node cluster using an unconventional method. Despite multiple attempts to resolve the issue, the problem […]

Problem: The client had a 10-node cluster across two data centers, with 5 nodes in each. They ran nodetool repair on one keyspace, but after 6 hours, it was stuck at 99% completion. They requested guidance on how to proceed with the issue, noting that their product version was ‘ReleaseVersion: 2.2.5.’ Process: Step 1 – […]

Problem: Cassandra v4.0.6 in non-production environments experienced an issue where the “cassandra” superuser password, which had been changed two months prior, reset to its default password (“cassandra”) after applying OS patches and rebooting the server. No manual password changes were evident in the audit logs. Solution: Initial Steps: The superuser “cassandra” initially had the default […]

Problem: The client needed to change the passwords of all users in the Cassandra cluster. It was specifically inquired about the necessity of changing the default password for the “cassandra” superuser and requested a step-by-step guide, along with precautions to prevent any impact on the application. Process: The expert provided a detailed response with the […]

Problem: The table is not updated immediately in Cassandra DB. The reference id table is not updated sometimes and the application gets old values. Process: Step 1: In the initial investigation and troubleshooting of the DB our experts asked the client the following questions: Cluster Configuration How many nodes are there in the cluster? What […]

Problem: The client reported that the Prometheus directory inside /var/lib had grown to 23GB, leading to high disk utilization on /var and potentially impacting other services. The /var directory has a total capacity of 200GB, which is shared by other service libraries and log files. Currently, the utilization on /var is at 80%, and the […]

Problem: The client, utilizing Kafka Streams application version 3.3.1, encountered issues with managing changelog topics. Despite configuring the application for automatic cleanup of records within these topics, unchecked growth was noticed, posing potential risks to system performance and stability. Process: Initial Assessment: The client reported the issue, highlighting the design of their Kafka Streams application […]

Problem: FreeIPA prompts regular users to change their passwords immediately after an admin resets them, which is undesired for certain admin-managed accounts like ‘admpass’. Process: The expert first reviewed the client’s IPA password policy and proposed using the krbPasswordExpiration attribute to control password expiration. However, attempts to set this attribute during user modification did not […]

Problem: The client experienced frequent warnings in their Cassandra nodes, indicated by the log entry: INFO [CompactionExecutor:37] 2023-02-23 12:00:12,268 NoSpamLogger.java:91 – Maximum memory usage reached (536870912), cannot allocate chunk of 1048576 The client requested an investigation into the potential impact of increasing the file_cache_size_in_mb in the cassandra.yaml file, whether a restart (bounce) would be necessary, […]